Sampling Error Estimation in Stratified Surveys

Many operations carried out by official statistical institutes use large-scale surveys obtained by stratified random sampling without replacement. Variables commonly examined in this type of surveys are binary, categorical and continuous, and hence, the estimates of interest involve estimates of proportions, totals and means. The problem of approximating the sampling relative error of this kind of estimates is studied in this paper. Some new jackknife methods are proposed and compared with plug-in and bootstrap methods. An extensive simulation study is carried out to compare the behavior of all the methods considered in this paper.


Introduction
Many of the operations carried out by official statistics institutions are based on surveys performed on a finite population using stratified random sampling without replacement.These surveys provide information on three types of variables: binary, categorical with more than two modalities, and continuous quantitative variables.Usually, most of the variables are binary, i.e. they yield two possible responses commonly coded with 1 and 0, and the aim is to estimate   1 p P  .A typical example is the variable indicating the presence or absence of a particular characteristic of interest in the study population, e.g. if a person is employed or unemployed, a household has Internet access or not, etc.In the case of categorical variables with more than two possible responses, say i , the aim is to estimate the proportion of each of the answers, , 1,  , , 2 For continuous quantitative variables, X , the objective is to estimate the mean of the variable, .
  Stratified sampling is an appropriate method when several homogeneous and mutually exclusive strata or subpopulations are identified in the population.Stratification can contribute to improve the representativeness of the sample by reducing sampling error.The bigger the differences between the strata, the greater the gain in accuracy.Moreover, some strata can be occasionally small in size but big in importance in the study.In these cases, an exhaustive sampling is recommended, i.e. all the individuals of these strata will be part of the sample.
These strata are called self-represented.
Once that a population quantity  , such as a mean or a proportion, has been estimated, it is important to obtain an accurate estimate of the sampling error to assess the reliability of our estimator ˆ .The sampling error of  can be presented in absolute terms, using the standard deviation of  , namely or in relative terms, using the variation coefficient of the estimator given by Var ˆ. rel E E

   
This work addresses the problem of estimating sampling errors in surveys carried out by stratified sampling from finite populations.Some new proposals to obtain sampling relative error estimates are introduced and various methods based on plug-in, jackknife and bootstrap techniques are compared.This research originated from the collaboration with the Galician Official Statistical Institute (IGE), the agency in charge of the official statistics in the Autonomous Community of Galicia, a region in the Northwest of Spain.Specifically, IGE was interested in assessing different criteria to approximate the sampling relative error for a variety of statistics produced by the Annual Company Survey (ACS), which is conducted by IGE in Galicia.About 200 variables are recorded in the Galician ACS, a survey including nearly 4000 companies in a total population of 65,000.The sample is obtained by stratified random sampling, with strata having different weights and some of them being self-represented.The strata are constructed by regarding the combination of three variables: the level of research performed in the company, the size of the company, measured in terms of the number of paid employees, and the main activity of the company.
The rest of this work is divided into the following sections.Notation used some basic concepts are introduced in Section 2. Section 3 describes the different jackknife procedures used to estimate the relative error of the estimators.The bootstrap methods of the error estimation are described in Section 4. Some results from a broad simulation study carried out to compare the behaviour of all the methods studied are presented in Section 5. Finally, the main conclusions of the work are established in Section 6.

Notation and Basic Concepts
From now on, and i will denote the population size, the number of strata and the population size of the i-th stratum, , respectively.Given a variable of interest, x is the j-th element of the i-th population stratum, with i .When the variable in study is quantitative (continuous or discrete), it will make sense to speak about the population mean, and the population total, being the sample size within the i -th stratum.Denoting by i i i F N n  the elevation factors of each stratum, the unbiased estimators of the mean  and the total  are obtained as follows. where Note that, by definition, i i for the strata selfrepresented in the sample, and hence the elevation factor of these strata is i . The unbiasedness property of  and  as esti- mators of  and  , respectively, follows from its con- struction as convex linear combinations of sample means (see (2.3) and (2.4)).So, from (2.1) and (2.2) follows Some simple calculations, although long (see, for example, [1]), yield the variance of these estimators.Specifically, and 2 1 ˆVar Var ,  being the variance of the finite population in the i-th stratum, given as Replacing the population variances i  , in (2.5) and (2.6) by the corresponding sample variances corrected by their degrees of freedom, ˆ we obtain plug-in estimators for the variance of  and  .Specifically, Var .
In the particular case of a binary variable we have where i denotes the true proportion of ones in the -th stratum and i the corresponding sampling proportion.Using the new expressions for i  and i S in (2.6) and (2.7), respectively, we obtain the variance of 2  and its plug-in estimator for binary variables, which are reduced to Going back to the general case in (2.5), it is deduced that the absolute and relative sampling errors of estimator  take the form Hence, the plug-in estimators of these errors are the relative errors of  and  as well as of their estimations are the same.Hence, the rest of the study will focus on the estimation of the relative error of parameter  , the population total.

Jackknife Estimation
The jackknife method is a general estimation procedure introduced by [2] that has been widely used to estimate the bias and the standard error of a statistic.It is well known that the jackknife technique leads to a reduction in the bias.Furthermore, the jackknife method is basically a resampling procedure, and hence one can estimate the accuracy of an estimator without assuming previous hypotheses on the distribution of the population.Let  be an estimator of a parameter  based on the sample 1 2 n X , , , then the jackknife estimator of the variance of  is defined as  is the jackknife pseudovalue, that is the estimator calculated using the whole sample except for the i-th observation, 1 , and X   is the mean of the jackknife pseudovalues, In a stratified sampling, the jackknife pseudovalues can be constructed following one of the two possible criteria: either removing a sample value at each iteration or removing a stratum at each iteration.Application of these criteria leads to two different jackknife estimators for the variance of  and  .Moreover, according to (2.4),  can be expressed as a linear combination of independent statistics   i X  , each one being separately constructed from the subsample of each stratum.Consequently, the variance of  can be calculated as a linear combination of variances of statistics constructed at stratum level.If these variances are previously estimated by jackknife in each stratum, then there will be a third way of using the jackknife to approximate the variance of  .Each of the three jackknife proposals are described more in detail below.

Jackknife Leaving One Sample Value out
Each jackknife pseudovalue is constructed by removing a single data value from the overall sample X , 1, , , 1, , Thus, the pseudovalue obtained when eliminating the s-th observation of the r-th stratum, X , takes the form .
On the other hand we have that By averaging all the pseudovalues, we obtain and the first variant of the jackknife estimator for the relative error is

Jackknife Leaving One Stratum out
Here we propose to calculate each pseudovalue removing all the observations of one stratum.Thus, the -th jackknife pseudovalue is based on the original sample without the observations of stratum , i.e.
Now, two variants of the jackknife estimator are introduced by considering different ways of averaging the pseudovalues   r  .First, we use a weighted mean, where each pseudovalue is weighted by the population size of the stratum removed in the calculation.Thus, we have Note that, in this case, it seems that a more simplified expression cannot be achieved.The jackknife estimator of the relative error is then calculated as An alternative variant of the jackknife leaving a stratum out is obtained if all the strata contribute with the same weight in the estimation, i.e. the pseudovalues are directly averaged as follows not admitting a simpler explicit expression either.The jackknife estimator of the relative error with this criterion is

Jackknife within Each Stratum
According to (2.4), the variance of  can be expressed as a linear combination of the variances of the sample means within each stratum Hence, a new jackknife approximation to the variance of   can be obtained by estimating each  Var i X  with the jackknife method and replacing these estimators in (3.4).For the jackknife estimator of

 
Var i X  , the pseudovalues are defined as , and their mean is given by  Then, the jackknife estimator of the variance of the sample mean of the i-th stratum is and using these previous jackknife estimations we obtain


The corresponding jackknife estimation of the relative error is given by 1

Bootstrap Estimation
An alternative resampling method often used to estimate the variance and the relative sampling error of the estimators is the bootstrap or self-sufficient estimation method.As any resampling procedure, including jackknife, the bootstrap takes advantage of not requiring hypotheses on the underlying distribution.The bootstrap method was introduced by Efron (see [3][4][5]) and has been widely treated in the literature.The basic idea of bootstrap consists in estimating the underlying population and then drawing out a number of resamples from the estimated population.By extracting a large number of these resamples (in the order of one or several thousand), bootstrap replications of a particular estimator  can be obtained and used to approximate the estimator variance and relative error.Âccording to (2.4), the estimator  can be written as the sum of the independent random variables , 1, , Hence, the bootstrap can be either applied to the global population, in order to directly estimate the variance of  , or to each of the strata, in order to estimate the variance of each statistic i i F X  and then approximate the variance of  as the sum of the variances of i i F X  .In particular, any bootstrap resampling plan ensuring the independence among different strata is valid to be used on the whole sample or stratum by stratum, although the latter is indeed more efficient computationally.Note also that the statistic i i F X  will have zero variance in self-represented strata, where the sampling is exhaustive i , and therefore it is not necessary to draw out bootstrap resamples in these strata.

 
1 F  i A detailed description of two bootstrap resampling procedures to approximate the population total in a fixed stratum , with , is provided below.

Bickel and Freedman Bootstrap Method
The proposal by Bickel and Freedman (see [6]) consists in estimating the underlying population from a mixture of two distinct and equal-sized finite populations.If i F is an integer number, then the bootstrap algorithm proceeds as follows.
BF.1 The estimated subpopulation for the i-th stratum is constructed by grouping i F identical copies of the sample of the stratum, that is the empirical population is given by Copyright © 2013 SciRes.

OJS
When the elevation factor i F is not integer, Steps BF.1 and BF.2 are modified as follows.

BF.1' Consider
denotes the integer part function.Thus, , with and 1 .
Two empirical finite subpopulations i and i N K n r   r  Rn   are now considered for the i-th stratum, which are formed, respectively, by i K and  identical copies of the observed sample, that is i n , is selected randomly and without replacement from are equal to the expected quantities when the size of the bootstrap population is i .Notice that the first resampling algorithm is a particular case of the previous one.In fact, if i F is an integer, i.e.R i = 0, then and both algorithms become identical.

Booth, Butler and Hall Bootstrap Method
The bootstrap procedure proposed by Booth, Butler and Hall (see [7]) is based on completing the population i estimated in Step BF.1' by adding a random subsample of the observed sample.In this way, one avoids using the two finite bootstrap populations i  and i .More precisely, Steps BF.1' and BF.2' of the Bickel and Freedman algorithm are modified as follows.
and r , with .

i i i i i
The finite subpopulation estimated for the i-th stratum is given by  , where  is constructed and i is formed by a random subsample of size generated without replacement from . 1  BBH.2A bootstrap resample of size , is selected at random and without replacement from  .The bootstrap estimation of the variance of the statistic  is now obtained using (2.4) and the bootstrap approximations i    Var  X  calculated with one of the algorithms previously described.Specifically, the bootstrap relative error is given by Note that the bootstrap variance for an exhaustive stratum is equal to zero, and hence the sum in (4.2) can be restricted to the non-exhaustive strata.Thus, we can write This clearly reduces the computational effort because no bootstrap resamples are generated from self-represented strata.As the self-represented strata do not affect to the sampling error, corrected versions of the jackknife estimators were also constructed by omitting the pseudovalues associated with these strata.These corrected versions are referred as the original ones but adding the letter "C".Note that this correction is implicit by construction in the case of the plug-in and bootstrap estimators.

Simulation Study
Three different types of response variables were simulated in our experiments: Binary variables.The response is "1" or "0" and the , which was chosen to take values close to 0.50, 0.30, 0.15 and 0.05.

Multinomial variables. Variables taking four possible results denoted by 1 2 3
A A A and 4 A were considered.Here, the parameter of interest is the vector , , Different scenarios of high and low variability between the mean responses of the population strata were simulated.In all our experiments, the population size was data values, classified in strata so that two of these strata were self-represented in the sampling.The first experiments were carried out with a sample size .Thus, the ratio between the sample and population sizes mimics the one in the Galician ACS conducted by IGE and that initially motivated the present work.
Our first results come from an experiment with binary response.The specific parameter values used to generate the population and the sample are shown in Table 1.
Table 1 summarizes the main features of the experiment.Thus, for instance, stratum L1 is formed by 1200 observations taking two possible values, "0" and "1", that have been randomly generated from Bernoulli trials with 1 .Note that all the values i are around 0.5, but there is a high variability between them.The overall population consists of 5000 observations and satisfies that and .
Under this population design, samples of size were generated, and hence 1000 estimates  were obtained with each studied method.In the case of the bootstrap procedures, a set of bootstrap replicates was considered to compute each estimator.The behaviour of each estimation procedure was examined by using the following values:  measures the efficiency of each estimator  with respect to the plug-in estimator,

  ˆPI ˆ1
Effic   .Thus,  means that the considered estimator  presents better behavior than the plug-in estimator in terms of mean squared error.Results from our first experiment are shown in Table 2, where different scales have been used to obtain a more intuitive comparison.
Similar experiments were carried out using different  , where the new sample size in the sample sizes  i-th stratum, i 1, , L  n kn , was determined by i i    , with i being the sample size of the first experiment (see Table 1) and k = 0.25, 0.30, 2, 3 and 4. Excluded from this rule are the two self-represented strata for which obviously  obtained with these new sample sizes are presented in Tables 3 and 4, respectively.
New trials of our first experiment were run with different values of parameter .   .In particular, it is observed that RMSE takes the smallest values when p is close to 0 or 0.5. and worsened with increasing n .Jackknife 1 estimator (based on leaving out one sample datum) was competitive although yielded worse results than the jackknife 3 with small sample sizes.Both variants of the jackknife 2 estimator (based on leaving out one stratum) yielded much worse results for all the considered sample sizes.Therefore, it is not advisable to use the jackknife 2 estimators.In general, the worse behaviour of the jackknife-based estimators seems to be mainly due to the bias of the estimation, although for the jackknife 2 estimators poor results are also observed in terms of standard deviation.

N 
observations divided in L = 10 strata was simulated.Data forming the -th stratum were randomly generated from a multinomial distribution with parameter , where  denotes the probability of event i A in the -th stratum, j   , 1, 2,3, 4 . Different values of j  were selected to set up scenarios with high and low variability of the responses between strata.In all cases, the overall population, which is formed by bringing together all strata, presents the theoretical parameter vector , , , 0.58, 0.24, 0.12, 0.06  No substantial differences were observed between using the jackknife procedures and their corrected versions (.C), based on cancelling the exhaustive strata.Just a slight advantage for the corrected versions is observed. Similar conclusions are valid for the two scenarios of high and low variability of the response between different strata.In general, the plug-in estimator is the most efficient in both situations, although the differences between the plug-in and the other methods are smaller in the case of low variability.
 For small sample sizes   145 n  , both two bootstrap methods and jackknife 3 led to a greater efficiency than the plug-in method.
A total of  samples of size were randomly selected and each of them was used to construct an estimator The set of estimates obtained with each of the studied methods   r is then used to calculate the mean squared error and the efficiency of each procedure with respect to the plug-in method.Both quantities were simultaneously obtained for each estimated marginal component i and jointly for the estimated vector  .In particular, the global Results from this new simulation are shown in Tables 6 and 7.
Results from Tables 6 and 7 show that the conclusions derived from simulations with binary response are equally valid for the case of multiple response.
Finally, we focus on the case of continuous response variables.Thus, new experiments were carried out to examine the performance of the different estimation procedures with responses generated from uniform, normal and exponential distributions.The simulation plan was designed by following the same outline as in previous experiments.
For instance, some experiments with normal and exponential responses were carried out considering the features summarized in Table 8.Note that these parameters lead to a situation of high variability between data from different strata.Some outcomes derived from these experiments are presented in Tables 9 and 10, for the Gaussian case, and in Tables 11 and 12 Results in Tables 9-12 allow us to confirm that the different analyzed estimation procedures behaved as in the previous experiments.Moreover, an analogous behaviour was also observed with uniform response and with scenarios of low variability between strata.In short, the conclusions derived from our numerical study with binary or multinomial response can be extended to the case of continuous response variables, regardless of the generating probability distribution.amined in this type of surveys are binary, categorical and continuous, and hence, the estimates of interest involve estimates of proportions, totals and means.In this setting, several procedures to approximate the sampling relative error of this kind of estimates are proposed.Different estimation techniques are considered, including the natural estimation of plug-in type and more sophisticated methods based on jackknife and bootstrap methodologies.The behaviour of the different procedures proposed is examined and compared by means of an extensive simulation study.In general, the plug-in method presents good behaviour in all the analyzed situations, with the additional advantage of having a low computational cost.For small sample sizes, the jackknife estimator denoted by "jackknife 3", which is based on the prior application of the jackknife technique to each stratum, and the two bootstrap methods considered (particularly the bootstrap proposed by Booth, Butler and Hall) yield results similar

Final Conclusion
The present work deals with the problem of estimating the sampling relative error of point estimates derived from large sample surveys.It is specifically assumed that the survey's sampling design is the stratified random sampling without replacement, because this design is often considered in many surveys conducted by different official statistics institutions.Variables commonly ex-as those obtained with the plug-in estimator, and in some cases, even better.However, the estimators obtained with these methods have a higher (although acceptable) computational cost.
of i  ensures that the mean and the variance in the resampling of a bootstrap observation ij X  N An extensive simulation study was performed to compare the estimators of the relative error of the population mean, previous sections.Specifically, the following estimators  were computed: the plug-in estimator each stratum) given in (3.1), (3.2), (3.3) and (3.5), respectively; and the bootstrap estimators derived from (4.1) using the resampling algorithms by Bickel and Freedman (BF estimator) and by Booth, Butler and Hall (BBH estimator).

2 3 
our simulation study is addressed to analyze the case of multiple response.Specifically, it is assumed that there are four mutually exclusive and exhaustive results for the response variable, let us say 1 The jackknife 3 estimator (based on applying jackknife to each stratum) behaved similarly to the bootstrap estimators for small sample sizes  A A A 4 and A .As in the case of binary response, a population of  n 400 two strata were considered to be exhaustive in the sampling.The theoretical error for the simulated population is   0.05, 0.09, 0.15, 0.21 

Table 2 . Results for the experiment with binary response conducted under parameters in Table 1. Total population para- meters:
Measure