• Previous Article
    An evolutionary multiobjective method for low-rank and sparse matrix decomposition
  • BDIA Home
  • This Issue
  • Next Article
    Introduction: Special issue on computational intelligence methods for big data and information analytics
January  2017, 2(1): 1-21. doi: 10.3934/bdia.2017005

Selective further learning of hybrid ensemble for class imbalanced increment learning

1. 

School of Computer Science and Technology, University of Science and Technology of China

2. 

HeFei, AnHui 230027, China, Springfield, MO 65801-2604, USA

Published  September 2017

Incremental learning has been investigated by many researchers. However, only few works have considered the situation where class imbalance occurs. In this paper, class imbalanced incremental learning was investigated and an ensemble-based method, named Selective Further Learning (SFL) was proposed. In SFL, a hybrid ensemble of Naive Bayes (NB) and Multilayer Perceptrons (MLPs) were employed. For the ensemble of MLPs, parts of the MLPs were selected to learning from the new data set. Negative Correlation Learning (NCL) with Dynamic Sampling (DyS) for handling class imbalance was used as the basic training method. Besides, as an additive model, Naive Bayes was employed as an individual of the ensemble to learn the data sets incrementally. A group of weights (with the number of the classes as the length) are updated for every individual of the ensemble to indicate the 'confidence' of the individual learning about the classes. The ensemble combines all of the individuals by weighted average according to the weights. Experiments on 3 synthetic data sets and 10 real world data sets showed that SFL was able to handle class imbalance incremental learning and outperform a recently related approach.

Citation: Minlong Lin, Ke Tang. Selective further learning of hybrid ensemble for class imbalanced increment learning. Big Data & Information Analytics, 2017, 2 (1) : 1-21. doi: 10.3934/bdia.2017005
References:
[1]

A. Asuncion and D. Newman, Uci machine learning repository, 2007.Google Scholar

[2]

G. A. CarpenterS. GrossbergN. MarkuzonJ. H. Reynolds and D. B. Rosen, Fuzzy artmap: A neural network architecture for incremental supervised learning of analog multidimensional maps, IEEE Transactions on Neural Networks, 3 (1992), 698-713. doi: 10.1109/72.159059. Google Scholar

[3]

G. A. Carpenter, S. Grossberg and J. H. Reynolds, ARTMAP: Supervised Real-Time Learning and Classification of Nonstationary Data by a Self-Organizing Neural Network Elsevier Science Ltd. , 1991. doi: 10.1109/ICNN.1991.163370. Google Scholar

[4]

N. V. ChawlaN. Japkowicz and A. Kotcz, Editorial: Special issue on learning from imbalanced data sets, Acm Sigkdd Explorations Newsletter, 6 (2004), 1-6. Google Scholar

[5]

G. DitzlerM. D. Muhlbaier and R. Polikar, Incremental learning of new classes in unbalanced datasets: Learn?+?+?.UDNC, International Workshop on Multiple Classifier Systems, Multiple Classifier Systems, (2010), 33-42. doi: 10.1007/978-3-642-12127-2_4. Google Scholar

[6]

G. DitzlerR. Polikar and N. Chawla, An incremental learning algorithm for non-stationary environments and class imbalance, International Conference on Pattern Recognition, (2010), 2997-3000. doi: 10.1109/ICPR.2010.734. Google Scholar

[7]

Y. Freund and R. E. Schapire, A short introduction to boosting, Journal of Japanese Society for Artificial Intelligence, 14 (1999), 771-780. Google Scholar

[8]

L. FuH.-H. Hsu and J. C. Principe, Incremental backpropagation learning networks, IEEE Transactions on Neural Networks, 7 (1996), 757-761. Google Scholar

[9]

H. He and E. A. Garcia, Learning from imbalanced data, IEEE Transactions on Knowledge and Data Engineering, 21 (2009), 1263-1284. Google Scholar

[10]

H. Inoue and H. Narihisa, Self-organizing neural grove and its applications, IEEE International Joint Conference on Neural Networks, 2 (2005), 1205-1210. doi: 10.1109/IJCNN.2005.1556025. Google Scholar

[11]

N. Japkowicz and S. Stephen, The Class Imbalance Problem: A Systematic Study IOS Press, 2002.Google Scholar

[12]

N. Kasabov, Evolving fuzzy neural networks for supervised/unsupervised online knowledge-based learning, IEEE Transactions on Systems Man & Cybernetics Part B Cybernetics A Publication of the IEEE Systems Man & Cybernetics Society, 31 (2001), 902-918. doi: 10.1109/3477.969494. Google Scholar

[13]

M. LinK. Tang and X. Yao, Dynamic sampling approach to training neural networks for multiclass imbalance classification, IEEE Transactions on Neural Networks and Learning Systems, 24 (2013), 647-660. Google Scholar

[14]

Y. Liu and X. Yao, Simultaneous training of negatively correlated neural networks in an ensemble, IEEE Transactions on Systems Man & Cybernetics Part B Cybernetics A Publication of the IEEE Systems Man & Cybernetics Society, 29 (1999), 716-725. Google Scholar

[15]

F. L. MinkuH. Inoue and X. Yao, Negative correlation in incremental learning, Natural Computing, 8 (2009), 289-320. doi: 10.1007/s11047-007-9063-7. Google Scholar

[16]

M. MuhlbaierA. Topalis and R. Polikar, Incremental learning from unbalanced data, In Neural Networks, 2004. Proceedings. 2004 IEEE International Joint Conference on, IEEE, 2 (2004), 1057-1062. doi: 10.1109/IJCNN.2004.1380080. Google Scholar

[17]

M. MuhlbaierA. Topalis and R. Polikar, Learn++.mt: A new approach to incremental learning, Lecture Notes in Computer Science, 3077 (2004), 52-61. doi: 10.1007/978-3-540-25966-4_5. Google Scholar

[18]

M. D. Muhlbaier, A. Topalis and R. Polikar, Learn ++. nc: combining ensemble of classifiers with dynamically weighted consult-and-vote for efficient incremental learning of new classes, IEEE Transactions on Neural Networks 20 (2009), p152.Google Scholar

[19]

S. OzawaS. Pang and N. Kasabov, Incremental learning of chunk data for online pattern classification systems, IEEE Trans Neural Netw, 19 (2008), 1061-1074. doi: 10.1109/TNN.2007.2000059. Google Scholar

[20]

R. PolikarJ. ByorickS. Krause and A. Marino, Learn++: A classifier independent incremental learning algorithm for supervised neural networks, International Joint Conference on Neural Networks, (2002), 1742-1747. doi: 10.1109/IJCNN.2002.1007781. Google Scholar

[21]

R. PolikarL. UpdaS. S. Upda and V. Honavar, Learn++: an incremental learning algorithm for supervised neural networks, IEEE Transactions on Systems Man & Cybernetics Part C, 31 (2001), 497-508. doi: 10.1109/5326.983933. Google Scholar

[22]

M. Salganicoff, Tolerating concept and sampling shift in lazy learning using prediction error context switching, Artificial Intelligence Review, 11 (1997), 133-155. doi: 10.1007/978-94-017-2053-3_5. Google Scholar

[23]

M. C. SuJ. Lee and K. L. Hsieh, A new artmap-based neural network for incremental learning, Neurocomputing, 69 (2006), 2284-2300. doi: 10.1016/j.neucom.2005.06.020. Google Scholar

[24]

Y. SunM. S. Kamel and Y. Wang, Boosting for learning multiple classes with imbalanced class distribution, In Data Mining, 2006. ICDM'06. Sixth International Conference on, IEEE, (2006), 592-602. doi: 10.1109/ICDM.2006.29. Google Scholar

[25]

E. K. TangP. N. Suganthan and X. Yao, An analysis of diversity measures, Machine Learning, 65 (2006), 247-271. doi: 10.1007/s10994-006-9449-2. Google Scholar

[26]

K. TangM. LinF. L. Minku and X. Yao, Selective negative correlation learning approach to incremental learning, Neurocomputing, 72 (2009), 2796-2805. doi: 10.1016/j.neucom.2008.09.022. Google Scholar

[27]

W. X. WenH. Liu and A. Jennings, Self-generating neural networks, International Joint Conference on Neural Networks, 4 (2002), 850-855. Google Scholar

[28]

G. Widmer and M. Kubat, Effective learning in dynamic environments by explicit context tracking, In Machine learning: ECML-93, Springer, 667 (1993), 227-243. doi: 10.1007/3-540-56602-3_139. Google Scholar

[29]

J. R. Williamson, Gaussian artmap: A neural network for fast incremental learning of noisy multidimensional maps, Neural Networks, 9 (1996), 881-897. doi: 10.1016/0893-6080(95)00115-8. Google Scholar

show all references

References:
[1]

A. Asuncion and D. Newman, Uci machine learning repository, 2007.Google Scholar

[2]

G. A. CarpenterS. GrossbergN. MarkuzonJ. H. Reynolds and D. B. Rosen, Fuzzy artmap: A neural network architecture for incremental supervised learning of analog multidimensional maps, IEEE Transactions on Neural Networks, 3 (1992), 698-713. doi: 10.1109/72.159059. Google Scholar

[3]

G. A. Carpenter, S. Grossberg and J. H. Reynolds, ARTMAP: Supervised Real-Time Learning and Classification of Nonstationary Data by a Self-Organizing Neural Network Elsevier Science Ltd. , 1991. doi: 10.1109/ICNN.1991.163370. Google Scholar

[4]

N. V. ChawlaN. Japkowicz and A. Kotcz, Editorial: Special issue on learning from imbalanced data sets, Acm Sigkdd Explorations Newsletter, 6 (2004), 1-6. Google Scholar

[5]

G. DitzlerM. D. Muhlbaier and R. Polikar, Incremental learning of new classes in unbalanced datasets: Learn?+?+?.UDNC, International Workshop on Multiple Classifier Systems, Multiple Classifier Systems, (2010), 33-42. doi: 10.1007/978-3-642-12127-2_4. Google Scholar

[6]

G. DitzlerR. Polikar and N. Chawla, An incremental learning algorithm for non-stationary environments and class imbalance, International Conference on Pattern Recognition, (2010), 2997-3000. doi: 10.1109/ICPR.2010.734. Google Scholar

[7]

Y. Freund and R. E. Schapire, A short introduction to boosting, Journal of Japanese Society for Artificial Intelligence, 14 (1999), 771-780. Google Scholar

[8]

L. FuH.-H. Hsu and J. C. Principe, Incremental backpropagation learning networks, IEEE Transactions on Neural Networks, 7 (1996), 757-761. Google Scholar

[9]

H. He and E. A. Garcia, Learning from imbalanced data, IEEE Transactions on Knowledge and Data Engineering, 21 (2009), 1263-1284. Google Scholar

[10]

H. Inoue and H. Narihisa, Self-organizing neural grove and its applications, IEEE International Joint Conference on Neural Networks, 2 (2005), 1205-1210. doi: 10.1109/IJCNN.2005.1556025. Google Scholar

[11]

N. Japkowicz and S. Stephen, The Class Imbalance Problem: A Systematic Study IOS Press, 2002.Google Scholar

[12]

N. Kasabov, Evolving fuzzy neural networks for supervised/unsupervised online knowledge-based learning, IEEE Transactions on Systems Man & Cybernetics Part B Cybernetics A Publication of the IEEE Systems Man & Cybernetics Society, 31 (2001), 902-918. doi: 10.1109/3477.969494. Google Scholar

[13]

M. LinK. Tang and X. Yao, Dynamic sampling approach to training neural networks for multiclass imbalance classification, IEEE Transactions on Neural Networks and Learning Systems, 24 (2013), 647-660. Google Scholar

[14]

Y. Liu and X. Yao, Simultaneous training of negatively correlated neural networks in an ensemble, IEEE Transactions on Systems Man & Cybernetics Part B Cybernetics A Publication of the IEEE Systems Man & Cybernetics Society, 29 (1999), 716-725. Google Scholar

[15]

F. L. MinkuH. Inoue and X. Yao, Negative correlation in incremental learning, Natural Computing, 8 (2009), 289-320. doi: 10.1007/s11047-007-9063-7. Google Scholar

[16]

M. MuhlbaierA. Topalis and R. Polikar, Incremental learning from unbalanced data, In Neural Networks, 2004. Proceedings. 2004 IEEE International Joint Conference on, IEEE, 2 (2004), 1057-1062. doi: 10.1109/IJCNN.2004.1380080. Google Scholar

[17]

M. MuhlbaierA. Topalis and R. Polikar, Learn++.mt: A new approach to incremental learning, Lecture Notes in Computer Science, 3077 (2004), 52-61. doi: 10.1007/978-3-540-25966-4_5. Google Scholar

[18]

M. D. Muhlbaier, A. Topalis and R. Polikar, Learn ++. nc: combining ensemble of classifiers with dynamically weighted consult-and-vote for efficient incremental learning of new classes, IEEE Transactions on Neural Networks 20 (2009), p152.Google Scholar

[19]

S. OzawaS. Pang and N. Kasabov, Incremental learning of chunk data for online pattern classification systems, IEEE Trans Neural Netw, 19 (2008), 1061-1074. doi: 10.1109/TNN.2007.2000059. Google Scholar

[20]

R. PolikarJ. ByorickS. Krause and A. Marino, Learn++: A classifier independent incremental learning algorithm for supervised neural networks, International Joint Conference on Neural Networks, (2002), 1742-1747. doi: 10.1109/IJCNN.2002.1007781. Google Scholar

[21]

R. PolikarL. UpdaS. S. Upda and V. Honavar, Learn++: an incremental learning algorithm for supervised neural networks, IEEE Transactions on Systems Man & Cybernetics Part C, 31 (2001), 497-508. doi: 10.1109/5326.983933. Google Scholar

[22]

M. Salganicoff, Tolerating concept and sampling shift in lazy learning using prediction error context switching, Artificial Intelligence Review, 11 (1997), 133-155. doi: 10.1007/978-94-017-2053-3_5. Google Scholar

[23]

M. C. SuJ. Lee and K. L. Hsieh, A new artmap-based neural network for incremental learning, Neurocomputing, 69 (2006), 2284-2300. doi: 10.1016/j.neucom.2005.06.020. Google Scholar

[24]

Y. SunM. S. Kamel and Y. Wang, Boosting for learning multiple classes with imbalanced class distribution, In Data Mining, 2006. ICDM'06. Sixth International Conference on, IEEE, (2006), 592-602. doi: 10.1109/ICDM.2006.29. Google Scholar

[25]

E. K. TangP. N. Suganthan and X. Yao, An analysis of diversity measures, Machine Learning, 65 (2006), 247-271. doi: 10.1007/s10994-006-9449-2. Google Scholar

[26]

K. TangM. LinF. L. Minku and X. Yao, Selective negative correlation learning approach to incremental learning, Neurocomputing, 72 (2009), 2796-2805. doi: 10.1016/j.neucom.2008.09.022. Google Scholar

[27]

W. X. WenH. Liu and A. Jennings, Self-generating neural networks, International Joint Conference on Neural Networks, 4 (2002), 850-855. Google Scholar

[28]

G. Widmer and M. Kubat, Effective learning in dynamic environments by explicit context tracking, In Machine learning: ECML-93, Springer, 667 (1993), 227-243. doi: 10.1007/3-540-56602-3_139. Google Scholar

[29]

J. R. Williamson, Gaussian artmap: A neural network for fast incremental learning of noisy multidimensional maps, Neural Networks, 9 (1996), 881-897. doi: 10.1016/0893-6080(95)00115-8. Google Scholar

Figure 1.  The Pseudo-Code For SFL
Figure 2.  This is Table 1
Figure 3.  This is table2
Figure 4.  This is table3
Figure 5.  This is table4
Figure 6.  This is table5
Figure 7.  This is table6
Figure 8.  This is table7
Figure 9.  This is table8
Figure 10.  This is table9
Figure 11.  This is table10
Figure 12.  This is table11
Figure 13.  This is table12
[1]

Anupama N, Sudarson Jena. A novel approach using incremental under sampling for data stream mining. Big Data & Information Analytics, 2017, 2 (5) : 1-13. doi: 10.3934/bdia.2017017

[2]

Yang Wang, Yi-fu Feng. $ \theta $ scheme with two dimensional wavelet-like incremental unknowns for a class of porous medium diffusion-type equations. Numerical Algebra, Control & Optimization, 2019, 0 (0) : 0-0. doi: 10.3934/naco.2019027

[3]

Luis D'Alto, Martin Corless. Incremental quadratic stability. Numerical Algebra, Control & Optimization, 2013, 3 (1) : 175-201. doi: 10.3934/naco.2013.3.175

[4]

Inmaculada Leyva, Irene Sendiña-Nadal, Stefano Boccaletti. Explosive synchronization in mono and multilayer networks. Discrete & Continuous Dynamical Systems - B, 2018, 23 (5) : 1931-1944. doi: 10.3934/dcdsb.2018189

[5]

Pierre Gabriel, Hugo Martin. Steady distribution of the incremental model for bacteria proliferation. Networks & Heterogeneous Media, 2019, 14 (1) : 149-171. doi: 10.3934/nhm.2019008

[6]

Alexandre J. Chorin, Fei Lu, Robert N. Miller, Matthias Morzfeld, Xuemin Tu. Sampling, feasibility, and priors in data assimilation. Discrete & Continuous Dynamical Systems - A, 2016, 36 (8) : 4227-4246. doi: 10.3934/dcds.2016.36.4227

[7]

Y. Gong, X. Xiang. A class of optimal control problems of systems governed by the first order linear dynamic equations on time scales. Journal of Industrial & Management Optimization, 2009, 5 (1) : 1-10. doi: 10.3934/jimo.2009.5.1

[8]

Yuri Kogan, Zvia Agur, Moran Elishmereni. A mathematical model for the immunotherapeutic control of the Th1/Th2 imbalance in melanoma. Discrete & Continuous Dynamical Systems - B, 2013, 18 (4) : 1017-1030. doi: 10.3934/dcdsb.2013.18.1017

[9]

Esther Chigidi, Edward M. Lungu. HIV model incorporating differential progression for treatment-naive and treatment-experienced infectives. Mathematical Biosciences & Engineering, 2009, 6 (3) : 427-450. doi: 10.3934/mbe.2009.6.427

[10]

Victor Meng Hwee Ong, David J. Nott, Taeryon Choi, Ajay Jasra. Flexible online multivariate regression with variational Bayes and the matrix-variate Dirichlet process. Foundations of Data Science, 2019, 1 (2) : 129-156. doi: 10.3934/fods.2019006

[11]

Alan Beggs. Learning in monotone bayesian games. Journal of Dynamics & Games, 2015, 2 (2) : 117-140. doi: 10.3934/jdg.2015.2.117

[12]

Yangyang Xu, Wotao Yin, Stanley Osher. Learning circulant sensing kernels. Inverse Problems & Imaging, 2014, 8 (3) : 901-923. doi: 10.3934/ipi.2014.8.901

[13]

Nicolás M. Crisosto, Christopher M. Kribs-Zaleta, Carlos Castillo-Chávez, Stephen Wirkus. Community resilience in collaborative learning. Discrete & Continuous Dynamical Systems - B, 2010, 14 (1) : 17-40. doi: 10.3934/dcdsb.2010.14.17

[14]

Mauro Maggioni, James M. Murphy. Learning by active nonlinear diffusion. Foundations of Data Science, 2019, 0 (0) : 1-21. doi: 10.3934/fods.2019012

[15]

Jueyou Li, Guoquan Li, Zhiyou Wu, Changzhi Wu, Xiangyu Wang, Jae-Myung Lee, Kwang-Hyo Jung. Incremental gradient-free method for nonsmooth distributed optimization. Journal of Industrial & Management Optimization, 2017, 13 (4) : 1841-1857. doi: 10.3934/jimo.2017021

[16]

E. Audusse. A multilayer Saint-Venant model: Derivation and numerical validation. Discrete & Continuous Dynamical Systems - B, 2005, 5 (2) : 189-214. doi: 10.3934/dcdsb.2005.5.189

[17]

Aaron A. Allen, Scott W. Hansen. Analyticity and optimal damping for a multilayer Mead-Markus sandwich beam. Discrete & Continuous Dynamical Systems - B, 2010, 14 (4) : 1279-1292. doi: 10.3934/dcdsb.2010.14.1279

[18]

Emmanuel Audusse, Fayssal Benkhaldoun, Jacques Sainte-Marie, Mohammed Seaid. Multilayer Saint-Venant equations over movable beds. Discrete & Continuous Dynamical Systems - B, 2011, 15 (4) : 917-934. doi: 10.3934/dcdsb.2011.15.917

[19]

A. Özkan Özer, Scott W. Hansen. Uniform stabilization of a multilayer Rao-Nakra sandwich beam. Evolution Equations & Control Theory, 2013, 2 (4) : 695-710. doi: 10.3934/eect.2013.2.695

[20]

Heman Shakeri, Faryad Darabi Sahneh, Caterina Scoglio, Pietro Poggi-Corradini, Victor M. Preciado. Optimal information dissemination strategy to promote preventive behaviors in multilayer epidemic networks. Mathematical Biosciences & Engineering, 2015, 12 (3) : 609-623. doi: 10.3934/mbe.2015.12.609

 Impact Factor: 

Metrics

  • PDF downloads (10)
  • HTML views (146)
  • Cited by (0)

Other articles
by authors

[Back to Top]