# American Institute of Mathematical Sciences

2014, 4(3): 193-207. doi: 10.3934/naco.2014.4.193

## Convergence analysis of the weighted state space search algorithm for recurrent neural networks

 1 Department of Applied Mathematics, The Hong Kong Polytechnic University, Kowloon, Hong Kong 2 Department of Mathematics, Cleveland State University, Cleveland, OH 44115

Received  April 2013 Revised  July 2014 Published  September 2014

Recurrent neural networks (RNNs) have emerged as a promising tool in modeling nonlinear dynamical systems. The convergence is one of the most important issues of concern among the dynamical properties for the RNNs in practical applications. The reason is that the viability of many applications of RNNs depends on their convergence properties. We study in this paper the convergence properties of the weighted state space search algorithm (WSSSA) -- a derivative-free and non-random learning algorithm which searches the neighborhood of the target trajectory in the state space instead of the parameter space. Because there is no computation of partial derivatives involved, the WSSSA has a couple of salient features such as simple, fast and cost effective. In this study we provide a necessary and sufficient condition that required for the convergence of the WSSSA. Restrictions are offered that may help assure convergence of the of the WSSSA to the desired solution. The asymptotic rate of convergence is also analyzed. Our study gives insights into the problem and provides useful information for the actual design of the RNNs. A numerical example is given to support the theoretical analysis and to demonstrate that it is applicable to many applications.
Citation: Leong-Kwan Li, Sally Shao. Convergence analysis of the weighted state space search algorithm for recurrent neural networks. Numerical Algebra, Control & Optimization, 2014, 4 (3) : 193-207. doi: 10.3934/naco.2014.4.193
##### References:
 [1] A. F. Atiya and A. G. Parlos, New results on recurrent network training: Unifying the algorithms and accelerating convergence,, IEEE Transcations on Neural Networks, 11 (2000), 697. [2] R. A. Conn, K. Scheinberg and N. L. Vicente, Introduction to Derivative-free Optimization,, SIAM, (2009). doi: 10.1137/1.9780898718768. [3] L. Jin, N. Nikifork and M. M. Gupta, Absolute stability conditions for discrete-time neural networks,, IEEE Tranc. Neural Networks, 5 (1994), 954. [4] L. K. Li, Learning sunspot series dynamics by recurrent neural networks,, Advances in Data Mining and Modeling (eds. W. K. Ching and K. P. Ng), (2003), 107. [5] L. K. Li and S. Shao, Dynamic properties of recurrent neural networks and its approximations,, International Journal of Pure and Applied Mathematics, 39 (2007), 545. [6] L. K. Li, S. Shao and T. Zheleva, A state space search algorithm and its application to learn the short-term foreign exchange rates,, Applied Mathematical Sciences, 2 (2008), 1705. [7] L. K. Li, Sally Shao and K. F. Cedric Yiu, Nonlinear dynamical system modeling via recurrent neural networks and a weighted wtate space search algorithm,, Journal of Industrial and Management Optimization, 7 (2011), 385. doi: 10.3934/jimo.2011.7.385. [8] Q. Liu and J. Wang, Finite-time convergent recurrent neural network with a hard-liming activation function for constrained optimization with piecewise-linear objective functions,, IEEE Transactions on Neural Networks, 22 (2011), 601. [9] D. T. Mirikitani and N. Nikolaev, Recursive Bayesian recurrent neural networks for time-series modeling,, IEEE Transactions on Neural Networks, 2 (2010), 262. [10] Q. Song, On the weight convergence of Elman networks,, IEEE Transactions on Neural Networks, 21 (2010), 463. [11] X. Wang and E. K. Blum, Discrete-time versus continuous-time models of neural networks,, Journal of Computer and System Sciences, 45 (1992), 1. doi: 10.1016/0022-0000(92)90038-K. [12] X. Wang and H. Huang, Convergence Study in Extended Kalman Filter-based Training of Recurrent Neural Networks,, IEEE Trans. on Neural Networks, 22 (2011), 588. [13] L. Xu and W. Liu, A new recurrent neural network adaptive approach for host-gate way rate control protocol within intranets using ATM ABR service,, Journal of Industrial and Management Optimization, 1 (2005), 389. doi: 10.3934/jimo.2005.1.337. [14] F. Xu and Z. Yi, Convergence Analysis of a class of simplified background netural networks with subnetworks,, Neurocomputing, 74 (2011), 3877. [15] J. Yao and C. L. Tan, A case study on using neural networks to perform technical forecasting of forex,, Neural Computation, 34 (2000), 79. [16] K. F. C. Yiu, S. Wang, K. L. Teo and A. C. Tsoi, Nonlinear System modeling via knot-optimizing B-splines networks,, IEEE Transactions on Neural Networks, 12 (2001), 1013. [17] Y. Zhang and K. K. Tan, Convergence Analysis of Recurrent Neural Networks., Kluwer, (2004). doi: 10.1007/978-1-4757-3819-3. [18] L. Zhang and Z. Yi., Selectable and unselectable sets of neurons in recurrent neural networks with saturated piecewise linear transfer function,, IEEE Transactions on Neural Networks, 22 (2011), 1021.

show all references

##### References:
 [1] A. F. Atiya and A. G. Parlos, New results on recurrent network training: Unifying the algorithms and accelerating convergence,, IEEE Transcations on Neural Networks, 11 (2000), 697. [2] R. A. Conn, K. Scheinberg and N. L. Vicente, Introduction to Derivative-free Optimization,, SIAM, (2009). doi: 10.1137/1.9780898718768. [3] L. Jin, N. Nikifork and M. M. Gupta, Absolute stability conditions for discrete-time neural networks,, IEEE Tranc. Neural Networks, 5 (1994), 954. [4] L. K. Li, Learning sunspot series dynamics by recurrent neural networks,, Advances in Data Mining and Modeling (eds. W. K. Ching and K. P. Ng), (2003), 107. [5] L. K. Li and S. Shao, Dynamic properties of recurrent neural networks and its approximations,, International Journal of Pure and Applied Mathematics, 39 (2007), 545. [6] L. K. Li, S. Shao and T. Zheleva, A state space search algorithm and its application to learn the short-term foreign exchange rates,, Applied Mathematical Sciences, 2 (2008), 1705. [7] L. K. Li, Sally Shao and K. F. Cedric Yiu, Nonlinear dynamical system modeling via recurrent neural networks and a weighted wtate space search algorithm,, Journal of Industrial and Management Optimization, 7 (2011), 385. doi: 10.3934/jimo.2011.7.385. [8] Q. Liu and J. Wang, Finite-time convergent recurrent neural network with a hard-liming activation function for constrained optimization with piecewise-linear objective functions,, IEEE Transactions on Neural Networks, 22 (2011), 601. [9] D. T. Mirikitani and N. Nikolaev, Recursive Bayesian recurrent neural networks for time-series modeling,, IEEE Transactions on Neural Networks, 2 (2010), 262. [10] Q. Song, On the weight convergence of Elman networks,, IEEE Transactions on Neural Networks, 21 (2010), 463. [11] X. Wang and E. K. Blum, Discrete-time versus continuous-time models of neural networks,, Journal of Computer and System Sciences, 45 (1992), 1. doi: 10.1016/0022-0000(92)90038-K. [12] X. Wang and H. Huang, Convergence Study in Extended Kalman Filter-based Training of Recurrent Neural Networks,, IEEE Trans. on Neural Networks, 22 (2011), 588. [13] L. Xu and W. Liu, A new recurrent neural network adaptive approach for host-gate way rate control protocol within intranets using ATM ABR service,, Journal of Industrial and Management Optimization, 1 (2005), 389. doi: 10.3934/jimo.2005.1.337. [14] F. Xu and Z. Yi, Convergence Analysis of a class of simplified background netural networks with subnetworks,, Neurocomputing, 74 (2011), 3877. [15] J. Yao and C. L. Tan, A case study on using neural networks to perform technical forecasting of forex,, Neural Computation, 34 (2000), 79. [16] K. F. C. Yiu, S. Wang, K. L. Teo and A. C. Tsoi, Nonlinear System modeling via knot-optimizing B-splines networks,, IEEE Transactions on Neural Networks, 12 (2001), 1013. [17] Y. Zhang and K. K. Tan, Convergence Analysis of Recurrent Neural Networks., Kluwer, (2004). doi: 10.1007/978-1-4757-3819-3. [18] L. Zhang and Z. Yi., Selectable and unselectable sets of neurons in recurrent neural networks with saturated piecewise linear transfer function,, IEEE Transactions on Neural Networks, 22 (2011), 1021.
 [1] Leong-Kwan Li, Sally Shao, K. F. Cedric Yiu. Nonlinear dynamical system modeling via recurrent neural networks and a weighted state space search algorithm. Journal of Industrial & Management Optimization, 2011, 7 (2) : 385-400. doi: 10.3934/jimo.2011.7.385 [2] Zhigang Zeng, Tingwen Huang. New passivity analysis of continuous-time recurrent neural networks with multiple discrete delays. Journal of Industrial & Management Optimization, 2011, 7 (2) : 283-289. doi: 10.3934/jimo.2011.7.283 [3] Lee DeVille, Nicole Riemer, Matthew West. Convergence of a generalized Weighted Flow Algorithm for stochastic particle coagulation. Journal of Computational Dynamics, 2018, 0 (0) : 1-26. doi: 10.3934/jcd.2019003 [4] Yazheng Dang, Fanwen Meng, Jie Sun. Convergence analysis of a parallel projection algorithm for solving convex feasibility problems. Numerical Algebra, Control & Optimization, 2016, 6 (4) : 505-519. doi: 10.3934/naco.2016023 [5] Yan Tang. Convergence analysis of a new iterative algorithm for solving split variational inclusion problems. Journal of Industrial & Management Optimization, 2017, 13 (5) : 1-20. doi: 10.3934/jimo.2018187 [6] Matteo Bonforte, Jean Dolbeault, Matteo Muratori, Bruno Nazaret. Weighted fast diffusion equations (Part Ⅱ): Sharp asymptotic rates of convergence in relative error by entropy methods. Kinetic & Related Models, 2017, 10 (1) : 61-91. doi: 10.3934/krm.2017003 [7] Tohru Nakamura, Shinya Nishibata, Naoto Usami. Convergence rate of solutions towards the stationary solutions to symmetric hyperbolic-parabolic systems in half space. Kinetic & Related Models, 2018, 11 (4) : 757-793. doi: 10.3934/krm.2018031 [8] Matthias Gerdts, Martin Kunkel. Convergence analysis of Euler discretization of control-state constrained optimal control problems with controls of bounded variation. Journal of Industrial & Management Optimization, 2014, 10 (1) : 311-336. doi: 10.3934/jimo.2014.10.311 [9] Jinyan Fan, Jianyu Pan. On the convergence rate of the inexact Levenberg-Marquardt method. Journal of Industrial & Management Optimization, 2011, 7 (1) : 199-210. doi: 10.3934/jimo.2011.7.199 [10] Shahad Al-azzawi, Jicheng Liu, Xianming Liu. Convergence rate of synchronization of systems with additive noise. Discrete & Continuous Dynamical Systems - B, 2017, 22 (2) : 227-245. doi: 10.3934/dcdsb.2017012 [11] Yves Bourgault, Damien Broizat, Pierre-Emmanuel Jabin. Convergence rate for the method of moments with linear closure relations. Kinetic & Related Models, 2015, 8 (1) : 1-27. doi: 10.3934/krm.2015.8.1 [12] Andriy Bondarenko, Guy Bouchitté, Luísa Mascarenhas, Rajesh Mahadevan. Rate of convergence for correctors in almost periodic homogenization. Discrete & Continuous Dynamical Systems - A, 2005, 13 (2) : 503-514. doi: 10.3934/dcds.2005.13.503 [13] Lixin Xu, Wanquan Liu. A new recurrent neural network adaptive approach for host-gate way rate control protocol within intranets using ATM ABR service. Journal of Industrial & Management Optimization, 2005, 1 (3) : 389-404. doi: 10.3934/jimo.2005.1.389 [14] J. Frédéric Bonnans, Justina Gianatti, Francisco J. Silva. On the convergence of the Sakawa-Shindo algorithm in stochastic control. Mathematical Control & Related Fields, 2016, 6 (3) : 391-406. doi: 10.3934/mcrf.2016008 [15] Y. K. Lin, C. S. Chong. A tabu search algorithm to minimize total weighted tardiness for the job shop scheduling problem. Journal of Industrial & Management Optimization, 2016, 12 (2) : 703-717. doi: 10.3934/jimo.2016.12.703 [16] Jui-Pin Tseng. Global asymptotic dynamics of a class of nonlinearly coupled neural networks with delays. Discrete & Continuous Dynamical Systems - A, 2013, 33 (10) : 4693-4729. doi: 10.3934/dcds.2013.33.4693 [17] Pavel Krejčí, Songmu Zheng. Pointwise asymptotic convergence of solutions for a phase separation model. Discrete & Continuous Dynamical Systems - A, 2006, 16 (1) : 1-18. doi: 10.3934/dcds.2006.16.1 [18] Yong Duan, Jian-Guo Liu. Convergence analysis of the vortex blob method for the $b$-equation. Discrete & Continuous Dynamical Systems - A, 2014, 34 (5) : 1995-2011. doi: 10.3934/dcds.2014.34.1995 [19] Fabio Camilli, Claudio Marchi. On the convergence rate in multiscale homogenization of fully nonlinear elliptic problems. Networks & Heterogeneous Media, 2011, 6 (1) : 61-75. doi: 10.3934/nhm.2011.6.61 [20] Oleg Makarenkov, Paolo Nistri. On the rate of convergence of periodic solutions in perturbed autonomous systems as the perturbation vanishes. Communications on Pure & Applied Analysis, 2008, 7 (1) : 49-61. doi: 10.3934/cpaa.2008.7.49

Impact Factor: