 Open Journal of Statistics, 2012, 2, 408-414 http://dx.doi.org/10.4236/ojs.2012.24049 Published Online October 2012 (http://www.SciRP.org/journal/ojs) Regularities in Sequences of Observations Mahkame Megan Khoshyaran Economics Traffic Clinic (ETC), Paris, France Email: megan.khoshyaran@wanadoo.fr Received July 18, 2012; revised August 20, 2012; accepted September 2, 2012 ABSTRACT The objective of this paper is to propose an adjustment to the three methods of calculating the probability that regulari-ties in a sample data represent a systemic influence in the population data. The method proposed is called data profiling. It consists of calculating vertical and horizontal correlation coefficients in a sample data. The two correlation coeffi- cients indicate the internal dynamic or inter dependency among observation points, and thus add new information. This information is incorporated in the already established methods and the consequence of this integration is that one can conclude with certainty that the probability calculated is indeed a valid indication of systemic influence in the popula- tion data. Keywords: Systematic Influence; Theory of Large Samples; Analysis of the Variance Principle; Multiple Regression; Data Profiling; Vertical Correlation Coefficient; Horizontal Correlation Coefficient 1. Introduction Suppose that in a sequence of observations one observes a striking regularity; for example suppose that the values arrange themselves in an increasing or decreasing order of magnitude, or a maximum or a minimum is indicated. Many questions arise. Is the observed regularity a general phenomenon, or is it true only of the sequence of the data set sampled. Is the observed regularity due to the par- ticular sequence sampled or is it due to sampling from a random sequence. In other words, in recurrent sampling, is it reasonable to believe that approximately the same general results will occur. Is it the manner of sampling that creates artificial regularities. The occurrence of regu- larity in a data set that results from random sampling is highly improbable; thus regularity in a sample data is a justification for regarding regularity as a true representa- tive of the population data. The assumption is that unless the probability of random occurrence is small, there is no objective proof that there exists an actual regularity in the population data. To explore regularities in random sample data sets many researchers have made significant contributions, [1-6]. For example, assuming that the sequence of indi- vidual numerical values is available, they have applied various tests based on characteristics of a random se- quence. For example, they concluded that the number of maxima in a sequence of unrelated numbers is one-third of the number of data points. The deviation of any se- quence of data in any characteristic from what is as- sumed for a random sample of sequences implies that there is a systematic influence, the extent of which de- pends on the magnitude of deviation and the number of data points in a sample. In general, random sampling of data is not a sufficient criteria for proving systematic influence. It is shown that unless there are a large number of data points, the proof of the existence of systematic influence remains unresolved. Up to now, the attempts to determine the probability of getting a short sequence of terms having a strict appear- ance of regularity have proven to be rather misleading. Given the uncertainties researchers have modified the analysis of regularities in random samples. In the new approach a sequence of averages of groups of individual observations is obtained in a systematic way. For exam- ple, random samples are drawn any number of times. The averages of each sequence of data sample are calculated. These averages form a composite sequence that can be used in testing the systematic influence in samples. The statistical significance of such a sequence of averages can be determined by comparing the variance of the in- dividual observations in a random sample computed di- rectly with that calculated from the variances of the av- erages, [7-14]. This analysis of variance principle can be applied to a general case where the values of the inde- pendent variable are related to each of a number of cor- related independent variables. This is a problem of mul- tiple curvilinear correlation, where a sequence of aver- ages of the dependent variable is computed with respect to each independent variable and correlated to the con- stant values of the other independent variables, [15-17]. Copyright © 2012 SciRes. OJS M. M. KHOSHYARAN 409o and nA method of testing the statistical significances of each sequence of averages as well as the composite signifi-cance of all of the sequences is derived.  should be less than the sampling error. If this principle holds then, Cox employed the criterion of significance. If the error of standard deviation of o is Although the use of a sequence of averages is a logical approach this method is highly uncertain and in some cases inapplicable. Analysis of variance principle, and the multiple curvilinear correlation have a solid logic, they provide approximate indications of any systematic influence. The main shortcoming of these models is that they do not detect the source of variability in a data set. The focus of the three models is on the variability within and the correlation among averages in a sample. To ad-dress this shortcoming of the three approaches, a modifi-cation to these models is proposed. The modification consists of detecting the correlation among individual observations both within and across groups in a data sample, or in another word, data profiling. This aim is achieved by calculating vertical vnon and horizontal h correlation coefficients, and incorporating them in the calculations. The precise definitions of these vari-ables are given and the manner in which they are inte-grated into the three models are demonstrated in the fol-lowing sections. 2. The Approach Based on the Theory of Large Samples Commonly, the values of sequences in a data sample are averages of measurements or numbers grouped in some systematic fashion. There are many readily available methods that calculate the probability of such systematic grouping of data. These methods are based on calculating the variability between the averages and within the groups. These methods are extended to special cases where the regularities of a sequence are periodic, [7-14]. The method based on the theory of large numbers devel-oped by  consists of computing the standard deviation of the groups means multiplied by the square root of the number of observations nn, and the standard de-viation of the entire series in a data sample o. Let m = number of columns or groups, n = number of entries per column, sya group mean, and y = the grand mean, then the group standard deviation is given by the following: 2syymnnn (1) The sample standard deviation is calculated using the following equation: 2syymno (2) It is assumed that the difference in value between small, then the standard error of the ratio () should n). Cox assumes that if there are be proportional to (systematic influences, then the expression 1non should on the average equal zero given the theory of large samples, and the standard error 2nonm. Practice has shown that this method that is based on the theory of large samples is often inapplicable. One way to circumvent this problem is to introduce vertical and horizontal correlation coefficients. Correlation coeffi-cients show the variations between observations, and across groups. A minor change of notation is introduced. The ys that represented the group mean is modified to ,1,,yj mj to reflect the mean by group of the observations or vertical means. Horizontal means ,1,,iyin reflect observation means. Each obser-vation is represented by ,1,,; 1,,yi nj mij . The vertical v and horizontal h correlation coeffi-cients are calculated using the following formulas: .102.0for1, ,nij jiivnijiyyy yjmyy (3) and 102.0for1,,mji ijjhmjijyyy yinyy (4) If the ratio maxmaxvh is equal to one, then the indication is that each observation is related to the other, both within each column and across columns; in other words, there is evidence of systematic influence or sys-tematic regularity. On the other hand, if the ratio maxmaxvh is either less than one or greater than one, then the evidence points to the contrary, which translates into the lack of any systematic influence. Thus, in general if the ratio non is equal to ma xmaxvh, equal to one, then there is absolute certainty that the po- Copyright © 2012 SciRes. OJS M. M. KHOSHYARAN 410 lation data exhibits systematic influence. The reverse case where the ratio non is equal to maxmaxvh,,,LAPR,;nj0yL, is either less than or greater than one, then there is no systematic influence in the population data. The approach based on the theory of large samples looks at the sample data from the macroscopic level, meaning sample averages and sample standard deviations. Data profiling explores the data set from the microscopic level, meaning the vertical and the horizontal correlation coefficients. Data profiling method adds new information which allows for an efficient and accurate detection of systemic influence. To state this formally, let be the space of almost surely random sets, where (Ω) is the set of all random sets, and (A) is a subset with σ-algebra. It can be stated that the sample data ij , and ij exhibits systemic influence if and only if the probability that 0,1,yi1,,m1mamaxmax and mapnnnnn00x;xVohVohLL exists and is equal to 1, or maxmaxVnhonPt00t0L, where (t) is some constant. Data profiling assigns to the space () a metric (d) which is associated with the probability of convergence. Let maxmaxmax,max max1maxVVnohndE nhoVnhonn0L; then it is easy to notice that (d) represents a distance in , and is invariant under any transformation (no matter which subset of random sets is used). If non and maxmaxvh are true representations of data at the two levels (macroscopic and microscopic respectively), then one would expect maxmaxvnohnd,0asn max 00maxvnhonPttif and only if . In fact the convergence of (d) to zero causes the con-vergence of the probability. This is due to the Bienaymé- Tchebychev or Markov inequality and the fact that as 1tnt t . Inversely if max 0maxvnhonPt holds then let for any (ε > 0) ;012ttt and 00max for, 1max 2vnhonPtnnn 0nn, then for maxmaxmaxmaxmax,maxmaxmax dmax1maxmaxmax dmax1maxmaxmax12 2vnhovnhovnohvnhovnnthovnhovnnthovnhondnPnnPnntPtt The conclusion is that maxmaxvh assures almost surely the detection of systemic influence in a data set. 3. The Approach Based on the Method of Analysis of Variance This method finds the probability that any variation in between averages is purely random, . An outline of the procedure follows: sn = number of entries in column (s) Copyright © 2012 SciRes. OJS M. M. KHOSHYARAN 411sNn is the total number of entries a = a reasonable estimate of y hya The mean variance between columns is calculated: 2211a Nhmn0mssjsnyV (5) The residual variance is calculated using the formula: 202ssny anmjryaVNm (6) Let log serVVZ, then the probability of no syste- matic influence is found from tables, [18-20] given (Z), and the degrees of freedom 1, and n2. The method of analysis of variance looks into the variability between column means and the variability of individual observa-tions from the corresponding mean within each column. This method has a shortcoming in that it does not look at the corresponding correlations between individual ob-servations in each column and across groups. Data pro-filing allows for a better analysis and detection of inter-nal or systematic variability. To account for data profil-ing, the formula for (Z) should be modified in the fol-lowing way: nmaxmaxvhlog logseerVZV. The addition of a log of the fraction of vertical and horizontal correlation coefficients has one major effect; it either augments the value of (Z), in which case lowers the probability of systematic influence or lowers the value of (Z), in which case raises the probability of sys-tematic influence. 4. The Approach Based on Multiple Regression Up to this point, we have been dealing with one inde- pendent variable only.  generalizes the method of analysis of variance to many independent variables which may be mutually correlated. In other words, the group averages are given as a multiple regression of (K) inde-pendent variables. He thus modifies the mean variance between columns sV, and the residual variance r, using the multiple regression method. The outline of the procedure is as follows: VM = Total number of columns (groups) to be averaged with respect to all the independent variables K = Number of independent variables Ky' = Value of an observation corrected with respect to all except the kth independent variable 1corrected group averages of the dependent variable.w.r.t. to independent variablesKsyKy 11weightedaverage of..weightedaverage ofsKKsyyyy The overall variance between columns is calculated:  222121200 01smm mKss ssssjj jVny ynyynyyKMK n   (7) The residual variance is calculated using the formula: 21ksrky yVNK Mn (8)  The probability that data being random is obtained as log serVZVbefore from , and the degrees of free- 1n, and 2n; and the probability of systematic dom 1logserVZVinfluence is thus . The shortcom- ing of the generalized method of the analysis of variance is that although it tries to look more closely at individual data sets, it does not look at the strength of the relation- ship between each individual data points. Data profiling in this case allows for adjusting for this shortcoming. The vertical and horizontal correlation coefficients, v, h are modified to adjust to the (K) independent variables. Let 1,,Kvv be the vertical correlation coe- fficients calculated for the K independent variables, and 1,,Khh be the horizontal correlation coefficients calculated for the K independent variables. New correla- tion coefficients are introduced: 1,,Krr 1,,Krrvv re- present residual vertical correlation coefficients of (Ky') adjusted observations, and hh repre- sent residual vertical correlation coefficients of (Ky') adjusted observations. For each independent variable (k), the vertical and horizontal correlation coefficients, v, h are calculated as before: (1)020for1,, ;and1, ,nkkk kiji jkivnkkijiyyy yjmyykK (9) Copyright © 2012 SciRes. OJS M. M. KHOSHYARAN right © 2012 SciRes. OJS 412 and 1020for1,, anmkkkkji ijjkhmkkjijyyy yinyyd1, ,kK (10) The overall variance between columns is then modified as follows:  1212maxmaxvvhh221212001201maxmaxmaxmaxmmss ssjjsKmvKss Kjhny ynyyVMK nny yKMK n    (11) In order to modify the residual variance, residual ver- tical and horizontal correlation coefficients are calculated using (ky'), the value of an observation which is cor- rected to constant values of all the rest except the kth independent variable given in McEwen’s generalized method of the analysis of variance, . The residual vertical correlation coefficients are calculated: Copy1020for1, ,;and1, ,nkkij jikivnkijiky ykyyrjmkKky y (12) and the residual horizontal correlation coefficients are given by: 102for 1,,mkkji ijjkhmkky y kyyr0and 1,,jijinkKky y (13) The residual variance is modified as: 2maxmax1kvkskhrky yrNK MnrV (14) The probability that data exhibits systematic influence log serZVVis obtained using 1n2and the degrees of free- dom (), and (n) as is already explained. 5. An Example: Sunspot Numbers In this section the validity of the improvement in the form of data profiling is tested. For this purpose the data set used in  is revisited and the probability of the existence of systematic influences in the data is calcu- lated once given the proposed analysis of variance method, which is already demonstrated in , and once with a modified version. Consider the data corresponding to sunspot numbers arranged with respect to a trial cycle of length 11 years, i.e. from 1749 to 1826. The sunspot numbers exceeding 99 are excluded. The data is shown in a matrix form as (Table 1): The averages sjyy are given: 52.5, 43.2, 26.0, 21.5, 13.5, 6.5, 7.5, 12.7, 24.5, 30.5, 43.2syn The number of columns is (m = 11). The number of observations in each column is (s = 4). The number of observations of the dependent variable is (N = 44). The overall average is y = 25.59. The degrees of freedom M. M. KHOSHYARAN 413Table 1. Sunspot numbers arranged with respect to a trail cycle of 11 years, 1749-1826.  1749-1759 1794-1804 1805-1815 1816-1826 1 81 41 42 46 2 83 21 28 41 3 48 16 10 30 4 48 6 8 24 5 31 4 3 16 6 12 7 0 7 7 10 15 1 4 8 10 34 5 2 9 32 45 12 9 10 48 43 14 17 11 54 48 35 36 1n, and 2n11 11044 1133 are respectively , and 2. The averages 1nnsy decrease up to the 6th column, and then increase from then on. To cal-culate the probability that the sample data is indicative of the population data, and thus there are cyclic effects, the (Z) statistic is calculated. The statistic (Z) is calculated using the mean variance between the columns sVV, and the residual variance (). rV956.32s, and 265.0rV. The statistic 956.32log 0.6408265.0serVZV. The value of (Z) corresponding to the 20, 5, 1, and 0.1 percent points are respectively 0.19, 0.38, 0.54, and 0.71. Since (Z = 0.64) is greater than 0.54, then the probability of random effects is 0.01, which makes the probability of systematic influence to be 0.99. Though the results seem to point in favor of systematic influence or the existence of cycles, the evidence is not conclusive. To find out if the sample obtained implies cyclic appearance of sun spots, the data profiling method is tested. The vertical and horizontal correlation coefficients are calculated given Equations (3) and (4). The vertical averages , 1,2,3,4yjj are calculated as: (41.5, 25jy.4, 14.3, 21.0v). The two statistics (), and (h) are calculated. 13.82, 5.40, 4.06, 1.41, 1.12, 1.85, 16.00, 12.22, 15.99v.13, 3.85, , 1322.41 (3280.54, 54.19, 1214.77h v) The max of (), and (h) are calculated as well. max v15.99605354.188689 max h The ratio maxmaxvh is calculated as: max 0.2951 917maxvh. The value maxlog maxveh is equal to 0.2586587. The modified value of the statistic (Z) is then obtained by adding the two values of maxloglog maxvseerhVZV     which then would give (0.6408 + 0.2587 ) = 0.8995. Since the value (0.8995) is higher than (0.71), it indicates that the probability that the population data is random is less than 0.001 which is less than 0.1 indicating with certainty that the number of sunspots is cyclic. The exis-tence of systemic influence is indisputable. Applying the approach based on the method of large samples, the 8nonstatistic is obtained. There is a large dis- crepancy between this statistic and the adjustment pro-posed in Section 2, max10.71maxvh. The statis-tic non is thus inapplicable. The statistic elog 0.6669srVZV calculated using the ap- proach based on multiple regression is a slight im- provement over the statistic obtained using the method of analysis of variance (Z = 0.6408). Using data profiling method, the statistic Z is corrected to (Z = 1.0). As in the case of the analysis of variance method, it can be stated with absolute certainty that there is indeed a systemic influence in the sample data. 6. Conclusion The objective is to derive conclusions about the random- ness of observations in a population given that the sam- ple data set exhibits strict regularities. Three methods are analyzed and their shortcomings are indicated. An im- provement to the three methods is suggested and formu- lated. The improvement comes in the form of data pro- filing which in essence is the integration of vertical and horizontal correlation coefficients in the equations. Through a simple example, it is shown that data profiling is indeed Copyright © 2012 SciRes. OJS M. M. KHOSHYARAN Copyright © 2012 SciRes. OJS 414 a compliment of the original formulation. REFERENCES  L. Besson, “On the Comparison of Methodological Data with Results of Chance,” Journal of Monthly Weather Review, Vol. 48, 1920, pp. 89-94.  H. W. Clough, “A Statistical Comparison of Meteoro- logical Data with Data of Random Occurrence,” Journal of Monthly Weather Review, Vol. 49, No. 3, 1921, pp. 124-132. doi:10.1175/1520-0493(1921)49<124:ASCOMD>2.0.CO;2  W. L. Crum, “A Measure of Dispersion for Ordered Se- ries,” Journal of American Statistical Association Quar- terly Publication, Vol. 17, 1921, pp. 969-975.  E. W. Wooland, “On the Mean Variability in Random Series,” Journal of Monthly Weather Review, Vol. 53, No. 3, 1925, pp. 107-111. doi:10.1175/1520-0493(1925)53<107:OTMVIR>2.0.CO;2  H. Working, “A Random Difference Series for Use in the Analysis of Time Series,” Journal of American Statistical Association Quarterly Publication, Vol. 24, 1934, pp. 11- 24. doi:10.1080/01621459.1934.10502683  W. O. Kermack and A. G. McKendrick, “A Measure of Dispersion for Ordered Series,” Journal of the Proceed- ings of the Royal Society Edinburgh, Vol. 57, 1937, pp. 228-240.  D. Alter, “A Group or Correlation Periodogram with Ap- plication to the Rainfall of the British Iles,” Journal of Monthly Weather Review, Vol. 55, No. 210, 1927, pp. 263-266. doi:10.1175/1520-0493(1927)55<263:AGOCPW>2.0.CO;2  C. Chree, “Periodicities Solar and Meteorological,” Jour- nal of the Royal Meteorological Society, Vol. 85, 1924, pp. 87-97.  J. B. Cox, “Periodic Fluctuations of Rainfall in Hawaii” Proceedings of the American Society of Civil Engineers, Vol. 87, 1924, pp. 461-491.  E. L. Dodd, “The Probability Law for the Intensity of a Trail Period with Data Subject to the Gaussian Law,” Bulletin of the American Mathematical Association Soci-ety, Vol. 33, 1927, pp. 681-684. doi:10.1090/S0002-9904-1927-04451-2  S. Kuznets, “Random Events and Cyclical Oscillations,” Journal of the American Statistical Association, Vol. 24, 1929, pp. 258-275. doi:10.1080/01621459.1929.10503048  R. W. Powell, “Successive Integration as a Method of Finding Long Period Cycles,” Annals of the Mathemati- cal Statistics, Vol. 1, No. 2, 1930, pp. 123-136. doi:10.1214/aoms/1177733127  K. Stumpff, “Grunlagen und Methoden der Periodenfor- schung,” Springer, Berlin, 1925.  G. T. Walker, “On Periodicity—Criteria for Reality,” Memorandum of the Royal meteorological Society, Vol. 3, No. 25, 1930, pp. 97-101.  C. F. McEwen and E. L. Michel, “The Functional Rela- tion of One Variable to Each of a Number of Correlated Variables Determined by a Method of Successive Ap- proximations to Group Averages,” Proceedings of the American Academy of Arts and Sciences, Vol. 55, No. 8, 1919, pp. 89-133.  C. F. McEwen, “The Minimum Temperature, a Function of the Dew Point and Humidity, at 5 p.m. of the Preced- ing Day; Method of Determining This Function by Suc- cessive Approximations to Group Averages,” Monthly Weather Review Supplement, No. 16, 1920, pp. 64-69.  C. F. McEwen, “The Reality of Regularities Indicated in Sequences of Observations,” Proceedings of the Berkeley Symposium on Mathematical Statistics and Probability, San Francisco, 13-18 August 1945, pp. 229-238.  R. A. Fisher, “Statistical Methods for Research Workers,” 4th Edition, Biological Monographs and Manuals, Lon- don, 1932.  G. U. Yule and M. G. Kendall, “An Introduction to the Theory of Statistics” 11th Edition, Charles Griffin and Company Ltd., London, 1937.  R. A. Fisher and F. Yates, “Statistical Tables for Biologi- cal, Agricultural, and Medical Research,” Oliver and Boyd, London, 1938.