Information Worth of MinMaxEnt Models for Time Series

In this study, by starting from Maximum entropy (MaxEnt) distribution of time series, we introduce a measure that quantifies information worth of a set of autocovariances. The information worth of autocovariences is measured in terms of entropy difference of MaxEnt distributions subject to different autocovariance sets due to the fact that the information discrepancy between two distributions is measured in terms of their entropy difference in MaxEnt modeling. However, MinMaxEnt distributions (models) are obtained on the basis of MaxEnt distributions dependent on parameters according to autocovariances for time series. This distribution is the one which has minimum entropy and maximum information out of all MaxEnt distributions for family of time series constructed by considering one or several values as parameters. Furthermore, it is shown that as the number of autocovariances increases, the entropy of approximating distribution goes on decreasing. In addition, it is proved that information worth of each model defined on the basis of MinMaxEnt modeling about stationary time series is equal to sum of all possible information increments corresponding to each model with respect to preceding model starting with first model in the sequence of models. The fulfillment of obtained results is demonstrated on an example by using a program written in Matlab.


Introduction
In many instances, the type of data available for modeling and that used for optimization is a set of observations measured over time of system variable(s) of interest [1]- [4].A time series stated as only one realization of a stochastic process is a set of data measured through time.In many areas from engineering to economics, patterns of time series are encountered.It is difficult to find a science program not required to study with a data set in form of time series.The characteristic property of a time series is that its future behavior can not be exactly estimated.It is not uncommon in economic analysis to develop a model and perform empirical analysis by assuming that economic agents make decisions based on a set of available information [5].In empirical analyses, however, the information is usually designated by a generic information set  .There is no attempt to quantify the amount of information in  .A quantification of the worth of such a set would not be an easy task even if one could identify all elements of  [6].In this paper, we view the flow of information to a stochastic process from the autocovariance sets and consider measuring the amount of information when  is a set which consists of autocovariances obtained from the time series.For this reason, it is concerned with the analysis of the ordered data using the principle of maximum entropy when the information about the times series is given by autocovariances up to a lag m.According to the maximum entropy approach, given time series can be viewed as single trial from a stochastic process that is stationary up to its second-order statistics and has a zero mean.It is known that MaxEnt distribution of an observed time series is determined as a multivariate normal distribution whose dimension is equal to the number of observations [1].By virtue of the entropy of normal distribution, entropy optimization (EO) functional is constructed as H max .It can be shown that as the number of constraints generated by autocovariances increases, value of H max decreases.In this investigation, firstly MaxEnt distribution for stationary time series subject to constraints generated by autocovariances set { } is considered.It is proved that as number of lags of successive autocovariances increases, the entropy value of this distribution goes on decreasing but its information worth goes on increasing.Furthermore, by starting from MaxEnt distribution dependent on parameters, MinMaxEnt distribution which has minimum entropy and maximum information out of all MaxEnt distributions is defined.It should be noted that MinMaxEnt and MaxMaxEnt distributions as solutions of Generalized Entropy Optimization (GEO) problem firstly are defined and generally investigated in [7]- [9].In [10], GEO distribution dependent on parameters in time series is introduced and via this distribution an estimation method of missing value is proposed.In this study, it is shown that entropy value and information worth of MinMaxEnt distribution obtained on the bases of MaxEnt distribution dependent on parameters has the same above expressed properties at each fixed value of parameters as MaxEnt distribution.In addition, it is proved that information worth of each model defined on the basis of MinMaxEnt modeling about stationary time series is equal to the sum of all possible information increments corresponding to each model with respect to preceding model starting with first model in the sequence of models.The fulfillment of obtained results is demonstrated on an example by the use of a program written in Matlab.

Information Worth of Autocovariances Set in MaxEnt Modeling
In this section, MaxEnt distributions according to different number of autocovariances are considered and it is proved that the entropy values of these distributions constitute a monotonically decreasing sequence when the number of autocovariances increases.Moreover it is shown that the information generated by autocovariances set is expressed as sum of information worth of each autocovariance taken separately. Theorem H r be the entropy value of this distribution.Then, entropy values of mentioned MaxEnt distributions form a monotonically decreasing sequence of the following form: Proof.The Shannon entropy measure subject to constraints generated by autocovariances 0 1 , , , m r r r  with 0,1, , m  of stationary time series { } 0 1 , , , N y y y  is multivariate normal [1].Therefore by increasing of the number of k of autocovariances vector { } { } 0 1 , , , , the conditions to maximize Shannon measure is increased and the domain of entropy measure becomes narrow.Consequently, entropy value of ( ) max k H r is strongly decreased and the inequalities (1) are satisfied.Theorem 1 is proved.
If we denote by ( ) k I r information worth of autocovariance r k , due to the fact that the information discrepancy between two distributions is measured in terms of their entropy difference in MaxEnt modeling, then Furthermore, if information worth generated by autocovariances set { } in the aggregate is denoted by ( ) Remark 1.The information ( ) is expressed as sum of information worths of each autocovariances taken separately, , , .

Information Worth of Dependent on Parameters
In this section, according to different number of autocovariances MaxEnt distributions dependent on parameters are considered and it is proved that at each value of parameter, these distributions and their entropies possess the same properties as in section 2. ) Then, information worth generated by autocovariances set ( ) .
Remark 2. The information  is expressed as sum of information worths of each autocovariances taken separately, ( , , . From ( 7) by virtue of formula ( 6) follows , , .

Information Worth of MinMaxEnt Models Dependent on Autocovariances
In this section, MinMaxEnt distributions (models) are obtained on the basis of MaxEnt distributions dependent on parameters and it is shown that as the number of autocovariances k goes on increasing, the entropy of approximating distribution (model) goes on decreasing.Furthermore, it is proved that information worth of each model defined on the basis of MinMaxEnt modeling about stationary time series is equal to the sum of all possible information to each model with respect to preceding model starting with first model in the sequence of models.
Then, between entropy values of MinMaxEnt distributions the inequalities ) ) From inequality (11) by taken into account (12) and (13), the inequality is got.If this process is consecutively repeated, then it is easy to get to the inequalities (10).Theorem 3 is proved.

Remark 3. By using Theorem 3, it is possible to obtain information worth of MinMaxEnt distributions with the different number of autocovariances.
By using Theorem 3, it is possible to obtain information worth of MinMaxEnt distributions with the different number of autocovariances.However, in order to simplify the description of results, we introduce the following symbols.Let k Y , ( ) be a model representing MinMaxEnt distribution .Moreover, let ( ) be the information contained by model k Y about this time series, then From ( 15) and ( 16), where k I is the information increment corresponding to each model k Y with respect to preceding model corresponding each model with respect to preceding model Y − starting with first model ( ) Equation (10) shows that as the number of autocovariances k increases, the entropy of approximating distribution (model) goes on decreasing but it never goes below the entropy of probability distribution satisfying the same conditions as MinMaxEnt distribution.According to (15) and (17) According to (18) in (19), 0

Applications
The developed MinMaxEnt models 0  Y − in the sense of information worth.On an example it is shown that mentioned estimated value is the best also in the sense of mean square error (MSE).To realize required operations, a program in MATLAB is written.For this purpose, we have considered data set generated from autoregressive process ( ) as follows: ( ) 1.90 2.01 1.84 6 0.80 , 0, 0.5 and the data set is given in Table 1.By using the data in Table 1, estimations based on MinMaxEnt models are obtained for missing values in each position via constraints generated by 0 1 2 , , r r r autocovariances and 0 1 2 3 , , , r r r r autocovariances.From Table 1 it is seen that, MinMaxEnt estimations 3t γ determined by the set consisting of 0 1 2 3 , , , r r r r autocovariances are better than MinMaxEnt estimations 2t γ determined by the set consisting of 0 1 2 , , r r r autocovariances in each position.Moreover, ( ) , , , r r r r MSE calculated by MinMaxEnt estimations with autocovariances is 0.2564 and it is lower than ( ) , r r autocovariances.Furthermore, in Table 2 the entropy and information worth of different autocovariance sets are given.These quantities calculated from the data set verify Theorem 4. It can be seen that as the number of constraints which is generated by autocovariances increases, the value of H max decreases.( )

Conclusions
In this study, the following results are established.
• MaxEnt distributions according to different number of autocovariances are considered and it is proved that the entropy values of these distributions constitute a monotonically decreasing sequence when the number of autocovariances increases.Moreover it is shown that the information generated by autocovariances set is expressed as sum of information worth of each autocovariance taken separately.• According to different number of autocovariances, MaxEnt distributions dependent on parameters are considered and it is proved that at each value of parameter these distributions and their entropies possess the same properties as the MaxEnt distributions.• MinMaxEnt distributions (models) are obtained on the basis of MaxEnt distributions dependent on parameters and it is shown that as the number of autocovariances k goes on increasing, the entropy of approximating distribution (model) goes on decreasing.Furthermore, it is proved that information worth of each model defined on the basis of MinMaxEnt modeling about stationary time series is equal to the sum of all possible information increments corresponding to each model with respect to preceding model starting with first model in the sequence of models.• Information worth of autocovariances in time series and values generating MinMaxEnt distributions can be applied in solving many problems.One of the mentioned problems is the problem of estimation of missing value in time series.It is proved that the value generating MinMaxEnt distribution independence on position represents the best estimation of the missing value in the sense of information worth.• The fulfillment of the obtained results is demonstrated on an example by using a program written in Matlab.
also dependent on γ .The- reafter, autocovariances set { } Thus, we have a family of time series dependent on γ .


In other words, entropy values of MaxEnt distributions dependent on γ constitute a monotonically decreas- ing sequence.Proof.According to Theorem 1, entropy values ( ) max k H r of MaxEnt distributions form a monotonically decreasing sequence of the form (1). Since ( ) depend on γ .Consequently, inequalities (5) are sa- tisfied.Theorem 2 is proved.Information worth of autocovariance r k dependent on γ is determined by the following equation similarly to (2),

)
are satisfied.Proof.According to Theorem 2 for any γ ,

Table 1 .
The data generated from AR(4) and its estimations with different autocovariance sets.

Table 2 .
Entropy and information worth of different autocovariance sets.