Information Worth of MinMaxEnt Models for Time Series

In this study, by starting from Maximum entropy (MaxEnt) distribution of time series, we introduce a measure that quantifies information worth of a set of autocovariances. The information worth of autocovariences is measured in terms of entropy difference of MaxEnt distributions subject to different autocovariance sets due to the fact that the information discrepancy between two distributions is measured in terms of their entropy difference in MaxEnt modeling. However, MinMaxEnt distributions (models) are obtained on the basis of MaxEnt distributions dependent on parameters according to autocovariances for time series. This distribution is the one which has minimum entropy and maximum information out of all MaxEnt distributions for family of time series constructed by considering one or several values as parameters. Furthermore, it is shown that as the number of autocovariances increases, the entropy of approximating distribution goes on decreasing. In addition, it is proved that information worth of each model defined on the basis of MinMaxEnt modeling about stationary time series is equal to sum of all possible information increments corresponding to each model with respect to preceding model starting with first model in the sequence of models. The fulfillment of obtained results is demonstrated on an example by using a program written in Matlab.

• Maximum Entropy Distribution
• Time Series
• Estimation of Missing Values
• MinMaxEnt Distribution
• Information Worth

1. Introduction

In many instances, the type of data available for modeling and that used for optimization is a set of observations measured over time of system variable(s) of interest [1] -[4] . A time series stated as only one realization of a stochastic process is a set of data measured through time. In many areas from engineering to economics, patterns of time series are encountered. It is difficult to find a science program not required to study with a data set in form of time series. The characteristic property of a time series is that its future behavior can not be exactly estimated. It is not uncommon in economic analysis to develop a model and perform empirical analysis by assuming that economic agents make decisions based on a set of available information [5] . In empirical analyses, however, the information is usually designated by a generic information set. There is no attempt to quantify the amount of information in. A quantification of the worth of such a set would not be an easy task even if one could identify all elements of [6] . In this paper, we view the flow of information to a stochastic process from the autocovariance sets and consider measuring the amount of information when is a set which consists of autocovariances obtained from the time series. For this reason, it is concerned with the analysis of the ordered data using the principle of maximum entropy when the information about the times series is given by autocovariances up to a lag m. According to the maximum entropy approach, given time series can be viewed as single trial from a stochastic process that is stationary up to its second-order statistics and has a zero mean. It is known that MaxEnt distribution of an observed time series is determined as a multivariate normal distribution whose dimension is equal to the number of observations [1] . By virtue of the entropy of normal distribution, entropy optimization (EO) functional is constructed as H_max. It can be shown that as the number of constraints generated by autocovariances increases, value of H_max decreases. In this investigation, firstly MaxEnt distribution for stationary time series subject to constraints generated by autocovariances set is considered. It is proved that as number of lags of successive autocovariances increases, the entropy value of this distribution goes on decreasing but its information worth goes on increasing. Furthermore, by starting from MaxEnt distribution dependent on parameters, MinMaxEnt distribution which has minimum entropy and maximum information out of all MaxEnt distributions is defined. It should be noted that MinMaxEnt and MaxMaxEnt distributions as solutions of Generalized Entropy Optimization (GEO) problem firstly are defined and generally investigated in [7] -[9] . In [10] , GEO distribution dependent on parameters in time series is introduced and via this distribution an estimation method of missing value is proposed. In this study, it is shown that entropy value and information worth of MinMaxEnt distribution obtained on the bases of MaxEnt distribution dependent on parameters has the same above expressed properties at each fixed value of parameters as MaxEnt distribution. In addition, it is proved that information worth of each model defined on the basis of MinMaxEnt modeling about stationary time series is equal to the sum of all possible information increments corresponding to each model with respect to preceding model starting with first model in the sequence of models. The fulfillment of obtained results is demonstrated on an example by the use of a program written in Matlab.

2. Information Worth of Autocovariances Set in MaxEnt Modeling

In this section, MaxEnt distributions according to different number of autocovariances are considered and it is proved that the entropy values of these distributions constitute a monotonically decreasing sequence when the number of autocovariances increases. Moreover it is shown that the information generated by autocovariances set is expressed as sum of information worth of each autocovariance taken separately.

Theorem 1. Let be autocovariances with lags of observed stationary time series, be MaxEnt distribution subject to constraints generated by autocovariances set ,; m < N and be the entropy value of this distribution. Then, entropy values of mentioned MaxEnt distributions form a monotonically decreasing sequence of the following form:

Proof. The Shannon entropy measure subject to constraints generated by autocovariances with of stationary time series is multivariate normal [1] . Therefore by increasing of the number of k of autocovariances vector, the conditions to maximize Shannon measure is increased and the domain of entropy measure becomes narrow. Consequently, entropy value of is strongly decreased and the inequalities (1) are satisfied. Theorem 1 is proved.

If we denote by information worth of autocovariance r_k, due to the fact that the information discrepancy between two distributions is measured in terms of their entropy difference in MaxEnt modeling, then

Furthermore, if information worth generated by autocovariances set in the aggregate is denoted by, , then

Remark 1. The information, , generated by autocovariances set is expressed as sum of information worths of each autocovariances taken separately,

From (3) by virtue of formula (2) follows

consequently

3. Information Worth of Dependent on Parameters

In this section, according to different number of autocovariances MaxEnt distributions dependent on parameters are considered and it is proved that at each value of parameter, these distributions and their entropies possess the same properties as in section 2.

Theorem 2. Let be MaxEnt distribution generated by autocovariances set of given stationary time series with missing value, where,. Since depend on, MaxEnt distribution is also dependent on. Thereafter, autocovariances set will be represented as, MaxEnt distribution as and entropy of this distribution as. Thus, we have a family of time series dependent on.

Between entropy values of MaxEnt distributions, the following inequalities are fulfilled:

In other words, entropy values of MaxEnt distributions dependent on constitute a monotonically decreasing sequence.

Proof. According to Theorem 1, entropy values of MaxEnt distributions form a monotonically decreasing sequence of the form (1). Since depend on. Consequently, inequalities (5) are satisfied. Theorem 2 is proved.

Information worth of autocovariance r_k dependent on is determined by the following equation similarly to (2),

Then, information worth generated by autocovariances set is denoted by , , and

Remark 2. The information, , generated by autocovariances set is expressed as sum of information worths of each autocovariances taken separately,

From (7) by virtue of formula (6) follows

4. Information Worth of MinMaxEnt Models Dependent on Autocovariances

In this section, MinMaxEnt distributions (models) are obtained on the basis of MaxEnt distributions dependent on parameters and it is shown that as the number of autocovariances k goes on increasing, the entropy of approximating distribution (model) goes on decreasing. Furthermore, it is proved that information worth of each model defined on the basis of MinMaxEnt modeling about stationary time series is equal to the sum of all possible information increments corresponding to each model with respect to preceding model starting with first model in the sequence of models.

Theorem 3. Let be MaxEnt distribution generated by autocovariances set of given stationary time series with parameter, at position s, where, and entropy value of this distribution be. Moreover, let, be the value realizing MinMaxEnt distribution, in other words

Then, between entropy values of MinMaxEnt distributions the inequalities

are satisfied.

Proof. According to Theorem 2 for any, , the inequalities (5) hold. For this reason,

On the other hand,

From inequality (11) by taken into account (12) and (13), the inequality

is got. If this process is consecutively repeated, then it is easy to get to the inequalities (10). Theorem 3 is proved.

Remark 3. By using Theorem 3, it is possible to obtain information worth of MinMaxEnt distributions with the different number of autocovariances.

By using Theorem 3, it is possible to obtain information worth of MinMaxEnt distributions with the different number of autocovariances. However, in order to simplify the description of results, we introduce the following symbols. Let, be a model representing MinMaxEnt distribution for a stationary time series. Moreover, let, be the information contained by model about this time series, then

and

From (15) and (16),

where is the information increment corresponding to each model with respect to preceding model. By virtue of the obtained results, the following theorem can be asserted.

Theorem 4. Information worth of model Y_m defined on the basis of MinMaxEnt modelling about stationary time series is equal to sum of all possible information increments , corresponding to each model with respect to preceding model starting with first model in the sequence of models.

Proof. By using the new notations inequalities (10) can be represented as

Equation (10) shows that as the number of autocovariances k increases, the entropy of approximating distribution (model) goes on decreasing but it never goes below the entropy of probability distribution satisfying the same conditions as MinMaxEnt distribution. According to (15) and (17)

or

According to (18) in (19), ,. Theorem 4 is proved.

5. Applications

The developed MinMaxEnt models can be applied to estimate the missing value in time series. According to Theorem 4, information worth generated by is greater than information worth generated by. Consequently, generating the model is the better estimation than generating the model in the sense of information worth. On an example it is shown that mentioned estimated value is the best also in the sense of mean square error (MSE). To realize required operations, a program in MATLAB is written. For this purpose, we have considered data set generated from autoregressive process as follows:

and the data set is given in Table 1. By using the data in Table 1, estimations based on MinMaxEnt models are obtained for missing values in each position via constraints generated by autocovariances and autocovariances. From Table 1 it is seen that, MinMaxEnt estimations determined by the set consisting of autocovariances are better than MinMaxEnt estimations determined by the set consisting of autocovariances in each position. Moreover, calculated by MinMaxEnt

estimations with autocovariances is 0.2564 and it is lower than calculated by MinMax-

Ent estimations with autocovariances and calculated by MinMaxEnt estimations with autocovariances.

Furthermore, in Table 2 the entropy and information worth of different autocovariance sets are given. These quantities calculated from the data set verify Theorem 4. It can be seen that as the number of constraints which is generated by autocovariances increases, the value of H_max decreases.

Table 1

The data generated from AR(4) and its estimations with different autocovariance sets

t				t
1	−7.6164	−3.7049	−4.5863	26	−2.5809	−0.4340	−2.2861
2	−7.9251	−5.9152	−8.2637	27	−1.8546	0.0249	−1.8094
3	−2.3466	−3.1335	−1.5912	28	4.7113	2.7239	4.5856
4	−1.0788	−2.6884	−1.1953	29	5.2406	2.9464	5.1481
5	−6.3050	−4.4728	−6.1961	30	−1.0943	0.4262	−0.8107
6	−7.7206	−4.9192	−7.7193	31	−2.4052	−0.1378	−2.3785
7	−2.2376	−2.6242	−2.2308	32	4.0709	3.0661	4.3309
8	0.33865	−1.6810	−0.0090	33	7.9505	4.9433	7.7333
9	−4.5611	−3.2248	−4.3121	34	3.5777	3.7644	3.5249
10	−7.3435	−4.7417	−7.6510	35	0.8252	3.1348	0.8623
11	−3.3723	−2.9111	−3.2169	36	−2.4052	−0.1378	−2.3785
12	0.13548	−1.8088	−0.0447	37	4.0709	3.0661	4.3309
13	−3.7786	−3.4259	−3.7174	38	7.9505	4.9433	7.7333
14	−8.2637	−5.3028	−8.1113	39	3.5777	3.7644	3.5249
15	−5.2458	−4.0749	−4.8305	40	0.8252	3.1348	0.8623
16	−0.2230	−2.2286	−0.1069	41	11.292	8.1259	11.159
17	−2.1272	−2.2977	−1.8858	42	7.5889	6.6536	7.3807
18	−5.4257	−2.6645	−5.2509	43	3.3139	5.2987	3.5224
19	−1.0920	−0.1997	−1.3106	44	6.5842	6.0319	6.2192
20	5.5526	3.1233	5.2295	45	10.412	7.5539	10.267
21	4.5110	3.1064	3.9525	46	7.2051	6.1065	7.1059
22	−0.8572	1.2503	−0.9899	47	2.0869	3.6081	2.1044
23	0.0716	1.1921	−0.2413	48	3.1468	3.5739	3.1619
24	4.7447	2.7488	4.8959	49	7.1153	5.0748	7.1544
25	3.4163	1.6973	3.3217	50	5.9239	4.4184	4.9659

Table 2

Entropy and information worth of different autocovariance sets

171.94	155.74	153.80	117.37	16.20	1.94	36.43	54.57

6. Conclusions

In this study, the following results are established.

・ MaxEnt distributions according to different number of autocovariances are considered and it is proved that the entropy values of these distributions constitute a monotonically decreasing sequence when the number of autocovariances increases. Moreover it is shown that the information generated by autocovariances set is expressed as sum of information worth of each autocovariance taken separately.

・ According to different number of autocovariances, MaxEnt distributions dependent on parameters are considered and it is proved that at each value of parameter these distributions and their entropies possess the same properties as the MaxEnt distributions.

・ MinMaxEnt distributions (models) are obtained on the basis of MaxEnt distributions dependent on parameters and it is shown that as the number of autocovariances k goes on increasing, the entropy of approximating distribution (model) goes on decreasing. Furthermore, it is proved that information worth of each model defined on the basis of MinMaxEnt modeling about stationary time series is equal to the sum of all possible information increments corresponding to each model with respect to preceding model starting with first model in the sequence of models.

・ Information worth of autocovariances in time series and values generating MinMaxEnt distributions can be applied in solving many problems. One of the mentioned problems is the problem of estimation of missing value in time series. It is proved that the value generating MinMaxEnt distribution independence on position represents the best estimation of the missing value in the sense of information worth.

・ The fulfillment of the obtained results is demonstrated on an example by using a program written in Matlab.

Acknowledgements

We thank the Editor and the referee for their comments. This support is greatly appreciated.

References

1. Kapur, J.N. and Kesavan, H.K. (1992) Entropy Optimization Principles with Applications. Academic Press, New York.

2. Wei, W.S. (2006) Time Series Analysis, Univariate and Multivariate Methods. Pearson, United States.

3. Box, G.E.P. and Jenkins, G. (1976) Time Series Analysis: Forecasting and Control. Holden-Day, United States.

4. Little, R. and Rubin, D. (1987) Statistical Analysis with Missing Data. Wiley, New York.

5. Pourahmadi, M. and Soofi, E. (1998) Prediction Variance and Information Worth of Observations in Time Series. Journal of Time Series Analysis, 21, 413-434. http://dx.doi.org/10.1111/1467-9892.00191

6. Pourahmadi, M. (1989) Estimation and Interpolation of Missing Values of a Stationary Time Series. Journal of Time Series Analysis, 10, 149-169. http://dx.doi.org/10.1111/j.1467-9892.1989.tb00021.x

7. Shamilov A. ,et al. (2006)A Development of Entropy Optimization Methods WSEAS Transaction on Mathematics 5, 568-575.

8. Shamilov, A. (2007) Generalized Entropy Optimization Problems and the Existence of Their Solutions. Physica A: Statistical Mechanics and its Applications, 382, 465-472. http://dx.doi.org/10.1016/j.physa.2007.04.014

9. Shamilov, A. (2010) Generalized Entropy Optimization Problems with Finite Moment Functions Sets. Journal of Statistics and Management Systems, 13, 595-603,. http://dx.doi.org/10.1080/09720510.2010.10701489

10. Shamilov, A. and Giriftinoglu, C. (2010) Generalized Entropy Optimization Distributions Dependent on Parameter in Time Series. WSEAS Transactions on Information Science and Applications, 1, 102-111.