Neural Modeling of Multivariable Nonlinear Stochastic System. Variable Learning Rate Case

The objective of this paper is to develop a variable learning rate for neural modeling of multivariable nonlinear stochastic system. The corresponding parameter is obtained by gradient descent method optimization. The effectiveness of the suggested algorithm applied to the identification of behavior of two nonlinear stochastic systems is demonstrated by simulation experiments.


Introduction
The Neural Networks (NN) was well used in modeling of nonlinear systems because of its ability of learning, its generalization and its approximation [1][2][3][4].Indeed, this approach provides an effective solution for wide classes of nonlinear systems which are not known or only partial state information is available [5].
Identification is the process of determining the dynamic model of a system from measurements inputs/ outputs [6].Often, the measured output system is tainted noise.This is due either to the effect of disturbances acting at different parts of the process, either to measurement noise.Therefore these noises may introduce errors in the identification.The stochastic model is a solution to overcome this problem [7].In this paper, a multivariable nonlinear stochastic system is our interest.
Among the parameters of the NN model, the learning rate ( )  has an important role in training phase.In this phase several tests are taken account to find the suitable fixed value.For instance, this parameter can slow down this phase of training [8,9] if it is small.However, if this parameter is large, the training phase is occurring quickly and it becomes unstable [8,9].To overcome this problem, an adaptive learning rate was asked in [8,9].This solution is applied in training algorithm of a nonlinear single-variable system [8] and in multivariable nonlinear system [9].In this paper, a variable learning rate of neural network is developed in order to model a multivariable nonlinear stochastic system.Different cases of signal ratio to noise (SNR) are taken account to show the influence of the noise in identification and the stability of training phase.
This paper is organized as follows.In second section, a multivariable system modeling by neural networks is presented.In third section, the fixed learning rate method is showed.The simulation of the multivariable stochastic systems by NN method using fixed learning rate is detailed in the fourth section.The development of the variable learning rate and results simulations are presented in fifth section.Conclusions are given in sixth section.

Multivariable System Modeling by Neural Networks
To find the neural model of such nonlinear systems, some stages must be respected [10].Firstly the input variables are standardized and centered.Then, the structure of the model is chosen.Finally, the synaptic weights are estimated and the obtained model must be validated.In this context, different algorithms are interested of the synaptic weights estimation.For instance, the gradient descent algorithm [11], the conjugate gradient algorithm [11], the one step secant [11], the Levenberg-Marquardt method [11] and resilient Backpropagation algorithm [11] are developed and confirmed their effectiveness in training.In this paper, the gradient descent algorithm is our interest.
On the basis of the input and output relation of a system, the above nonlinear system can be expressed by a NARMA (Nonlinear Auto-Regressive Moving Average) 1, N i l model [12], that is given by the Equation ( 1).The architecture of the RNN is presented in Figure 1.
The output of the hidden node is given by the following equation: The neural output is given by the following equation: Finally, the compact form is defined as: The principle of neural modeling of the multivariable stochastic system is showing in Figure 1.
To show the influence of disturbances on modeling, a noise signal is added to the output system.Different cases of Signal Noise Ratio ( i SNR ) are taken.This ( i SNR ) measures the correspondence between the system output and the estimated output, the equation of is as follows: The accuracy of correlations relative to the measured values is finding by various statistical means.The criteria exploited in this study were the Relative Error (RE), Root Mean Square Error (RMSE) and Mean Absolute Percentage Error (MAPE) [11] given by :

Fixed Learning Rate Method
The neural system modeling is the research of parameters (weights) model.The search of these weights is the subjects of different works [1][2][3][4][5][6][8][9][10][11][12][13].The gradient descent method is one among different methods which was well applied on neural identification for single-variable system [8] and for multivariable system [9].In this paper, the same principle is suggested to be applied on neural identification of the multivariable stochastic systems.Indeed, the criterion is minimized as follows: By application of the GD method, the theory of [1] is used; we find then [9]:  For the variation of the synaptic weights of the hidden layer towards the output layer with The compact form (4) is used here, so we find Finally, the synaptic weights of the hidden layer towards the output layer can be written in the following way:  For the variation of the synaptic weights of the input layer towards the hidden layer.
Finally, the synaptic weights of input layer towards the hidden layer can be written in the following way: In these expressions, i  is a positive constant value [8,9] which represents the learning rate (0 1)

Simulation of Multivariable Nonlinear Stochastic Systems ( 5) SNR  
In this section, two types of multivariable nonlinear stochastic systems with 2 dimensions are presented with .The system  [8] and [14] are defined respectively by the following equations: with 1 and 2 are a random signals, or 1 u and 2 are the input signals of the systems considered defined by: and The input signal and are presented in Figure 2.

Simulation Results of System (S1)
A dynamic NN is used to simulate a multivariable nonlinear stochastic system (S1) .In Figure 3, the evolution of the process output and the NN output of the system (S1) is presented.The estimation error between these two outputs is presented in Figure 4.

Simulation Results of System (S2)
A dynamic NN is used to simulate a multivariable nonlinear stochastic system (S2) .In Figure 5, the evolution of the process output and the NN output of the system (S2) is presented.The estimation error between these two outputs is presented in Figure 6.  2 for the first system and in table 3 for the second system.
In both Tables 2 and 3, when the increases the SNR

 
i mse e decrease, it is due under the presence of disturbances in the system.
In this section, the simulation of the two systems (S1 and S2) is carried out using a fixed learning rate.To find the suitable learning rate it is necessary to carry out several tests by keeping the condition that .This research of the learning rate can slow down the phase of training.To cure this disadvantage and in order to accelerate the phase of training, a variable learning rate is used and a fast algorithm will be developed.

The Proposed Fast Algorithm
The need for using a variable learning rate is to have a fast training [8][9][15][16][17][18]].To answer this condition, the difference of the estimation error at and at is calculated [8,9].
We suppose that by application of [8,9]     then the Equation (20) can be

o k o k e k f h p k
We introduce (10) and (12), so we find To ensure the convergence of the estimation error, be satisfied [8,9].This condition implies It is clear that the upper range of the learning rate ( ) is variable because


depends on , il and lj .The fastest learning occurs when the learning rate is: It's certain that the learning process cannot finish instantly because of the approximation which is caused by the finite sampling time contrary to the theory which is proved that it can be happen if infinitely fast sampling can occur.
Using the obtained variable learning rate i  , the synaptic weights il z  and lj p  will be respectively.it's certain that the learning process cannot finish instantly because of the approximation caused by the finite sampling time contrary to the theorie which proved that it can be happen if infinitely fast sampling can occur it's certain that the learning process cannot finish instantly because of the approximation caused by the finite sampling time contrary to the theorie which proved that it can be happen if infinitely fast sampling can occur it's certain that the learning process cannot finish instantly because of the approximation caused by the finite sampling time contrary to the theorie which proved that it can be happen if infinitely fast sampling can occur it's certain that the learning process cannot finish instantly because of the approximation caused by the finite sam-pling time contrary to the theorie which proved that it can be happen if infinitely fast sampling can occur.
Finally, and can be: rate, the neural output 1 follows the measured output are applied.In Figure 7, the evolution of the process output and the NN output of the system (S1) is presented.The error estimation between these two outputs is presented in Figure 8.

Simulation Results of System (S2) ( 5) SNR 
The obtained results present that for a variable learning The evolution of the process output and the NN output of the system (S2) is presented in Figure 9.The error between these two outputs is presented in Figure 10.The evolution of the squared error in two cases; fixed and variable learning rates is presented in Figures 11 and 12.The obtained results, concerning system (S2), present that for a variable learning rate, the neural output 1 follows the measured output with an error of predicttion  However, if and , the error of prediction is and [9].
The obtained results presented in Figures 11 and 12 showing that, when a variable learning rate is used, the convergence of the squared error is very faster than a fixed learning rate is used.
Table 4 shows the obtained results of each statistical indicator in the system (S1) and (S2) in the case of variable learning rate.
We took three cases of to show the influence of disturbances modeling.The obtained results are presented in Table 5 for the first system and in Table 6 for the second system.In both tables, when the increases the decrease, it is due under the presence of disturbances in the system.

Conclusions
In this paper, a variable learning rate for neural modeling of multivariable nonlinear stochastic system is suggested.This parameter can slow down the training phase when it is chosen as small, and can be unstable when it is chosen as large.To avoid this step, a variable learning rate method is developed and it is applied in identification of nonlinear stochastic system.The advantages of the proposed algorithm are firstly the simplicity to apply it in a multi-input multi-output nonlinear system.Secondly, the gain of the training time is remarked and the result quality is noticed.Besides, this algorithm is a manner to avoid the search for such fixed training rate which presents a disadvantage at the level the phase of training.In contrary, the variable learning rate algorithm does not require any experimentation for the selection of an appropriate value of the learning rate.The proposed algorithm can be applied in real time process modeling.Different cases of SNR are discussed to test the developed method and it showed that the obtained results using a variable learning rate is very satisfy than when the fixed learning rate was used.
, ns : number of nodes of output layer,

Figure 1 .
Figure 1.Principle of the neural modeling of the multivariable stochastic system.

Figure 2 .
Figure 2. Input signals of the multivariable nonlinear stochastic system.

Figure 3 .Figure 4 .
Figure 3. Output of process and NN of system (S1) using a fixed learning rate.

Figure 7 .
Figure 7. Output of process and NN of system (S1) using a variable learning rate.

Figure 8 .Figure 9 .Figure 10 .
Figure 8. Learning error between the output of process and NN.

1 N 2 N
: number of nodes of input layer, : number of nodes of hidden layer, lj p : synaptic weights of the input layer towards the hid- noise of measurement of symmetric terminal  , average value.