Stochastic Binary Neural Networks for Qualitatively Robust Predictive Model Mapping

A. T. Burrell, P. Papantoni-Kazakos

Computer Science Department, Oklahoma State University, Stillwater, USA.
Electrical Engineering Department, University of Colorado Denver, Denver, USA.

DOI: 10.4236/ijcns.2012.529070                

Abstract

We consider qualitatively robust predictive mappings of stochastic environmental models, where protection against outlier data is incorporated. We utilize digital representations of the models and deploy stochastic binary neural networks that are pre-trained to produce such mappings. The pre-training is implemented by a back propagating supervised learning algorithm which converges almost surely to the probabilities induced by the environment, under general ergodicity conditions.

Keywords

Qualitative Robustness; Predictive Model Mapping; Stochastic Approximation; Stochastic Binary Neural Networks; Real-Time Supervised Learning; Ergodicity

Conflicts of Interest

The authors declare no conflicts of interest.

References

[1]	F. Abdelhamid, “Transformation of Observations in Stochastic Approximation,” Annals of Statistics, Vol. 1, No. 6, 1973, pp. 1158-1174. Hdoi:10.1214/aos/1176342564
[2]	V. Fabian, “On Asymptotic Normality in Stochastic Approximation,” Annals of Mathematical Statistics, Vol. 39, No. 4, 1968, pp. 1327-1332. Hdoi:10.1214/aoms/1177698258
[3]	D. Kazakos and P. Papantoni-Kazakos, “Detection and Estimation,” Computer Science Press, New York, 1989.
[4]	J. Kiefer and J. Wolfowitz, “Stochastic Estimation of the Maximum of a Regression Function,” Annals of Mathematical Statistics, Vol. 23, No. 3, 1952, pp. 462-466. Hdoi:10.1214/aoms/1177729392
[5]	H. Kushner, “Asymptotic Global Behavior for Stochastic Approximations and Diffusions with Slowly Decreasing Noise Effects: Global Minimization via Monte Carlo,” SIAM Journal of Applied Mathematics, Vol. 47, No. 1, 1987, pp. 169-185. Hdoi:10.1137/0147010
[6]	H. Kushner and D. Clark, “Stochastic Approximation Methods for Constrained and Unconstrained Systems,” Springer-Verlag, Berlin, 1978. Hdoi:10.1007/978-1-4684-9352-8
[7]	L. Ljung, “Analysis of Recursive Stochastic Algorithms,” IEEE Transactions on Automatic Control, Vol. 22, No. 4, 1977, pp. 551-575. Hdoi:10.1109/TAC.1977.1101561
[8]	L. Ljung and T. S?derstr?m, “Theory and Practice of Recursive Identification,” MIT Press, Cambridge, 1983.
[9]	T. Y. Young and R. A. Westerberg, “Stochastic Approximation with a Nonstationary Regression Function,” IEEE Transactions on Information Theory, Vol. IT-18, No. 4, 1972, pp. 518-519. Hdoi:10.1109/TIT.1972.1054851
[10]	R. Beran, “Adaptive Autoregressive Process,” Annals of the Institute of Statistical Mathematics, Vol. 28, No. 1, 1976, pp. 77-89. Hdoi:10.1007/BF02504731
[11]	J. R. Blum, “Multidimensional Stochastic Approximation Procedure,” Annals of Mathematical Statistics, Vol. 22, No. 4, 1954, pp. 737-744. Hdoi:10.1214/aoms/1177728659
[12]	R. A. Fisher, “The Goodness of Fit of Regression Formulae and the Distribution of Regression Coefficients,” Journal of the Royal Statistical Society, Vol. 85, No. 4, 1922, pp. 597-612. Hdoi:10.2307/2341124
[13]	L. Gerencser, “Parameter Tracing of Time-Varying Continuous-Time Linear Stochastic Systems,” In: C. I. Byrnes and A. Lindquist, Eds., Modeling, Identification and Robust Controls, North-Holland, Amsterdam, 1986, pp. 581-594.
[14]	R. L. Kashyap and C. C. Blaydon, “Recovery of Functions from Noisy Measurements Taken at Randomly Selected Points and Its Application to Pattern Classification,” Proceedings of the IEEE, Vol. 54, No. 8, 1966, pp. 1127-1129. Hdoi:10.1109/PROC.1966.5051
[15]	R. L. Kashyap, C. Blaydon and K. S. Fu, “Stochastic Approximation,” In: J. M. Mendel and K. S. Fu, Eds., Adaptive Learning and Pattern Recognition Systems, Academic Press, New York, 1970, pp. 329-355. Hdoi:10.1016/S0076-5392(08)60499-3
[16]	H. Robbins and S. Monro, “A Stochastic Approximation Method,” Annals of Mathematical Statistics, Vol. 22, No. 3, 1951, pp. 400-407. Hdoi:10.1214/aoms/1177729586
[17]	A. R. Barron, F. W. van Straten and R. L. Barron, “Adaptive Learning Network Approach to Weather Forecasting: A Summary,” Proceedings of the International Conference on Cybernetics and Society, 1977, pp. 724-727.
[18]	J. Elman and D. Zipser, “Learning the Hidden Structure of Speech,” Journal of the Acoustical Society of America, Vol. 83, No. 4, 1988, pp. 1615-1626. Hdoi:10.1121/1.395916
[19]	P. Gorman and T. Sejnowski, “Analysis of Hidden Units in a Layered Network Trained to Classify Sonar Targets,” Neural Networks, Vol. 1, No. 1, 1988, pp. 75-90. Hdoi:10.1016/0893-6080(88)90023-8
[20]	M. Minsky and S. Papert, “Perceptrons,” MIT Press, Cambridge, 1969.
[21]	F. Rosenblatt, “The Perceptron: A Perceiving and Recognizing Automaton,” Report 85-60-1, Cornell Aeronautical Laboratory, Buffalo, New York, 1957.
[22]	P. Werbos, “Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences,” Ph.D. Dissertation, Harvard University, 1974.
[23]	H. White, “Some Asymptotic Results for Learning in Single Hidden-Layer Feedforward Network Models,” American Statistical Association, Vol. 84, No. 408, 1989, pp. 1003-1013. Hdoi:10.1080/01621459.1989.10478865
[24]	B. Widrow, “Generalization and Information Storage in Networks of Adaline Neurons,” In: M. D. Yovits, G. T. Jacobi and G. D. Goldstein, Eds., Self-Organizing Systems, Spartan Books, Washington DC, 1962, pp. 435-461.
[25]	B. Widrow and M. E. Hoff, “Adaptive Switching Circuits,” 1960 IRE WESCON Convention Record, 1960, pp. 96-104.
[26]	R. E. Blahut, “Hypothesis Testing and Information Theory,” IEEE Transactions on Information Theory, Vol. IT-20, 1987, pp. 405-417.
[27]	D. H. Ackley, G. E. Hinton and T. J. Sejnowski, “A Learning Algorithm for Boltzman Machines,” Cognitive Science, Vol. 9, No. 1, 1985, pp. 147-169. Hdoi:10.1207/s15516709cog0901_7
[28]	S. Amari, K. Kurata and H. Nagoaka, “Information Geometry of Boltzman Machines,” IEEE Transactions on Neural Networks, Vol. 3, No. 2, 1992, pp. 260-271. Hdoi:10.1109/72.125867
[29]	D. Pados and P. Papantoni-Kazakos, “New Non-Least-Squares Neural Network Learning Algorithms for Hypothesis Testing,” IEEE Transactions on Neural Networks, Vol. 6, No. 3, 1995, pp. 596-609. Hdoi:10.1109/72.377966
[30]	D. Pados, K. W. Halford, D. Kazakos and P. Papantoni- Kazakos, “Distributed Binary Hypothesis Testing with Feedback,” IEEE Transactions on Systems, Man, and Cybernetics, Vol. 25, No. 1, 1995, pp. 21-42.
[31]	Pados, P. Papantoni-Kazakos D. Kazakos and A. Kogiantis, “On-Line Threshold Learning for Neyman-Pearson Distributed Detection,” IEEE Transactions on Systems, Man, and Cybernetics, Vol. 24, No. 10, 1994, pp. 1519-1531. Hdoi:10.1109/21.310534
[32]	D. Pados and P. Papantoni-Kazakos, “A Note on the Estimation of the Generalization Error and the Prevention of Overfitting,” IEEE International Conference on Neural Networks, Orlando, 1994.
[33]	D. Pados and P. Papantoni-Kazakos, “A Class of Neyman-Pearson and Bayes Learning Algorithms for Neural Classification,” IEEE International Symposium on Information Theory, Trondheim, 1-27 July 1994.
[34]	A. G. Kogiantis and P. Papantoni-Kazakos, “Operations and Learning in Neural Networks for Robust Prediction,” IEEE Transactions on Systems, Man, and Cybernetics, Vol. 27, No. 3, Part B, 1997, pp. 402-411.
[35]	Y. Liu, Z. Wang and X. Liu, “Robust Stability of Discrete-Time Stochastic Neural Networks with Time-Varying Delays,” 4th International Symposium on Neural Networks, Vol. 71, No. 4-6, 2008, pp. 823-833.
[36]	Z. Wang, Y. Liu, M. Li and X. Liu, “Stability for Stochastic Cohen-Grossberg Neural Networks with Mixed Time Delays,” IEEE Transactions on Neural Networks, Vol. 17, No. 3, 2006, pp. 814-820. Hdoi:10.1109/TNN.2006.872355
[37]	H. Ling, “Stochastic Neural Networks,” LAP Lambert Academic Publishing, 2010.
[38]	P. Papantoni-Kazakos, D. Kazakos and K. Birmiwal, “Predictive Analog-to-Digital Conversion for Resistance to Data Outliers,” Information and Computation, Vol. 98, No. 1, 1992, pp. 56-98. Hdoi:10.1016/0890-5401(92)90042-E