# American Institute of Mathematical Sciences

February 2019, 2(1): 43-53. doi: 10.3934/mfc.2019004

## Eliminating other-race effect for multi-ethnic facial expression recognition

 1 Dalian Key Lab of Digital Technology for National Culture, College of Computer Science and Engineering, Dalian Minzu University, Dalian 116600, Liaoning, China 2 Department of Computing, Curtin University, Kent Street, Perth, WA 6102, Australia

* Corresponding author: Xiaodong Duan

Published  March 2019

Fund Project: This work is supported by National Natural Science Foundation of China (Grant No.61672132), Science and Technology Foundation of Liaoning Province of China (Grant No.20170520234)ìNational Natural Science Foundation of China (Grant No.61602321), and Natural Science Fund Project of Liaoning Province (No.20170540694).

It has been noticed that the performance of multi-ethnic facial expression recognition is affected by other-race effect significantly. Though this phenomenon has been noticed by psychologists and computer vision researchers for decades, the mechanism of other-race effect is still unknown and few work has been done to compensate or remove this effect. This work proposes an ICA-based method to eliminate the other-race effect in automatic 3D facial expression recognition. Firstly, the depth features are extracted from 3D local facial patches, and independent component analysis is applied to project the features into a subspace in which the projected features are mutually independent. The ethnic-related features and expression-related features are supposed to be separated in ICA subspace. Hence, ethnic-sensitive features are then determined by an entropy-based feature selection method and discarded to depress their influence on facial expression recognition. The proposed method is evaluated on benchmark BU-3DFE database, and the experimental results reveal that the influence caused by other-race effect can be suppressed effectively with the proposed method.

Citation: Mingliang Xue, Xiaodong Duan, Wanquan Liu. Eliminating other-race effect for multi-ethnic facial expression recognition. Mathematical Foundations of Computing, 2019, 2 (1) : 43-53. doi: 10.3934/mfc.2019004
##### References:
 [1] B. M. Craig, Z. Jing and O. V. Lipp, Facial race and sex cues have a comparable influence on emotion recognition in chinese and australian participants, Attention Perception and Psychophysics, 79 (2017), 2212-2223. doi: 10.3758/s13414-017-1364-z. [2] M. N. Dailey, G. W. Cottrell, C. Padgett and R. Adolphs, Empath: A neural network that categorizes facial expressions, Journal of Cognitive Neuroscience, 14 (2002), 1158-1173. doi: 10.1162/089892902760807177. [3] M. N. Dailey, C. Joyce, M. J. Lyons, M. Kamachi, H. Ishi, J. Gyoba and G. W. Cottrell, Evidence and a computational explanation of cultural differences in facial expression recognition, Emotion, 10 (2010), 874-893. doi: 10.1037/a0020019. [4] C. Darwin and I. J. Rachman, The Expression Of Emotions In Man And Animals, Julian Friedmann, 1979. [5] J. R. D'Errico, Understanding gridfit, Information available at: http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do. [6] P. Ekman and W. V. Friesen, Facial action coding system (facs): A technique for the measurement of facial actions, Rivista Di Psichiatria, 47 (1978), 126-138. [7] P. Ekman, E. R. Sorenson and W. V. Friesen, Pan-cultural elements in facial displays of emotion, Science, 164 (1969), 86-88. doi: 10.1126/science.164.3875.86. [8] G. A. Feingold, The influence of environment on identification of persons and things, Journal of the American Institute of Criminal Law and Criminology, 5 (1914), 39-51. doi: 10.2307/1133283. [9] S. Fu, H. He and Z.-G. Hou, Learning race from face: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 36 (2014), 2483-2509. doi: 10.1109/TPAMI.2014.2321570. [10] A. Hyvarinen, Fast and robust fixed-point algorithms for independent component analysis, IEEE Transactions on Neural Networks, 10 (1999), 626-634. doi: 10.1109/72.761722. [11] R. E. Jack, C. Blais, C. Scheepers, P. G. Schyns and R. Caldara, Cultural confusions show that facial expressions are not universal, Current Biology Cb, 19 (2009), 1543-1548. doi: 10.1016/j.cub.2009.07.051. [12] R. E. Jack, O. G. B. Garrod, H. Yu, R. Caldara and P. G. Schyns, Facial expressions of emotion are not culturally universal, Proceedings of the National Academy of Sciences of the United States of America, 109 (2012), 7241-7244. doi: 10.1073/pnas.1200155109. [13] H. Li, J. Sun and L. Chen, Location-sensitive sparse representation of deep normal patterns for expression-robust 3d face recognition, in Biometrics (IJCB), 2017 IEEE International Joint Conference on, IEEE, 2017, 234-242. doi: 10.1109/BTAS.2017.8272703. [14] H. Li, J. Sun, Z. Xu and L. Chen, Multimodal 2d+ 3d facial expression recognition with deep fusion convolutional neural network, IEEE Transactions on Multimedia, 19 (2017), 2816-2831. doi: 10.1109/TMM.2017.2713408. [15] R. S. Malpass and J. Kravitz, Recognition for faces of own and other race, Journal of Personality and Social Psychology, 13 (1969), 330-334. doi: 10.1037/h0028434. [16] V. Natu and A. J. O'Toole, Neural perspectives on the other-race effect, Visual Cognition, 21 (2013), 1081-1095. doi: 10.1080/13506285.2013.811455. [17] P. J. Phillips, F. Jiang, A. Narvekar, J. Ayyad and A. J. O'Toole, An other-race effect for face recognition algorithms, ACM Transactions on Applied Perception (TAP), (2010), 1-12. doi: 10.6028/NIST.IR.7666. [18] J. A. Russell, Cross-cultural similarities and differences in affective processing and expression, in Emotions and Affect in Human Factors and Human-Computer Interaction, Elsevier, 2017, 123-141. doi: 10.1016/B978-0-12-801851-4.00004-5. [19] J. M. Susskind, D. H. Lee, A. Cusi, R. Feiman, W. Grabski and A. K. Anderson, Expressing fear enhances sensory acquisition, Nature Neuroscience, 11 (2008), 843-850. doi: 10.1038/nn.2138. [20] M. Wang, W. Deng, J. Hu, J. Peng, X. Tao and Y. Huang, Racial faces in-the-wild: Reducing racial bias by deep unsupervised domain adaptation, arXiv preprint, arXiv: 1812.00194. [21] M. Xue, X. Duan, J. Zhou, C. Wang, Y. Wang, Z. Li and W. Liu, A computational other-race-effect analysis for 3d facial expression recognition, Chinese Conference on Biometric Recognition, 2016, 483-493. doi: 10.1007/978-3-319-46654-5_53. [22] M. Xue, A. Mian, W. Liu and L. Li, Fully automatic 3d facial expression recognition using local depth features, in IEEE Winter Conference on Applications of Computer Vision, IEEE, 2014, 1096-1103. [23] X. Yan, T. J. Andrews, R. Jenkins and A. W. Young, Cross-cultural differences and similarities underlying other-race effects for facial identity and expression, Quarterly Journal of Experimental Psychology, 69 (2016), 1247-1254. doi: 10.1080/17470218.2016.1146312. [24] L. Yin, X. Wei, Y. Sun, J. Wang and M. J. Rosato, A 3d facial expression database for facial behavior research, in 7th International Conference on Automatic Face and Gesture Recognition (FGR06), IEEE, 2006, 211-216. [25] Q. Zhen, D. Huang, Y. Wang and L. Chen, Muscular movement model based automatic 3d facial expression recognition, in International Conference on Multimedia Modeling, Springer, 2015, 522-533. doi: 10.1007/978-3-319-14445-0_45. [26] Q. Zhen, D. Huang, Y. Wang and L. Chen, Muscular movement model-based automatic 3d/4d facial expression recognition, IEEE Transactions on Multimedia, 18 (2016), 1438-1450. doi: 10.1109/TMM.2016.2557063.

show all references

##### References:
 [1] B. M. Craig, Z. Jing and O. V. Lipp, Facial race and sex cues have a comparable influence on emotion recognition in chinese and australian participants, Attention Perception and Psychophysics, 79 (2017), 2212-2223. doi: 10.3758/s13414-017-1364-z. [2] M. N. Dailey, G. W. Cottrell, C. Padgett and R. Adolphs, Empath: A neural network that categorizes facial expressions, Journal of Cognitive Neuroscience, 14 (2002), 1158-1173. doi: 10.1162/089892902760807177. [3] M. N. Dailey, C. Joyce, M. J. Lyons, M. Kamachi, H. Ishi, J. Gyoba and G. W. Cottrell, Evidence and a computational explanation of cultural differences in facial expression recognition, Emotion, 10 (2010), 874-893. doi: 10.1037/a0020019. [4] C. Darwin and I. J. Rachman, The Expression Of Emotions In Man And Animals, Julian Friedmann, 1979. [5] J. R. D'Errico, Understanding gridfit, Information available at: http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do. [6] P. Ekman and W. V. Friesen, Facial action coding system (facs): A technique for the measurement of facial actions, Rivista Di Psichiatria, 47 (1978), 126-138. [7] P. Ekman, E. R. Sorenson and W. V. Friesen, Pan-cultural elements in facial displays of emotion, Science, 164 (1969), 86-88. doi: 10.1126/science.164.3875.86. [8] G. A. Feingold, The influence of environment on identification of persons and things, Journal of the American Institute of Criminal Law and Criminology, 5 (1914), 39-51. doi: 10.2307/1133283. [9] S. Fu, H. He and Z.-G. Hou, Learning race from face: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 36 (2014), 2483-2509. doi: 10.1109/TPAMI.2014.2321570. [10] A. Hyvarinen, Fast and robust fixed-point algorithms for independent component analysis, IEEE Transactions on Neural Networks, 10 (1999), 626-634. doi: 10.1109/72.761722. [11] R. E. Jack, C. Blais, C. Scheepers, P. G. Schyns and R. Caldara, Cultural confusions show that facial expressions are not universal, Current Biology Cb, 19 (2009), 1543-1548. doi: 10.1016/j.cub.2009.07.051. [12] R. E. Jack, O. G. B. Garrod, H. Yu, R. Caldara and P. G. Schyns, Facial expressions of emotion are not culturally universal, Proceedings of the National Academy of Sciences of the United States of America, 109 (2012), 7241-7244. doi: 10.1073/pnas.1200155109. [13] H. Li, J. Sun and L. Chen, Location-sensitive sparse representation of deep normal patterns for expression-robust 3d face recognition, in Biometrics (IJCB), 2017 IEEE International Joint Conference on, IEEE, 2017, 234-242. doi: 10.1109/BTAS.2017.8272703. [14] H. Li, J. Sun, Z. Xu and L. Chen, Multimodal 2d+ 3d facial expression recognition with deep fusion convolutional neural network, IEEE Transactions on Multimedia, 19 (2017), 2816-2831. doi: 10.1109/TMM.2017.2713408. [15] R. S. Malpass and J. Kravitz, Recognition for faces of own and other race, Journal of Personality and Social Psychology, 13 (1969), 330-334. doi: 10.1037/h0028434. [16] V. Natu and A. J. O'Toole, Neural perspectives on the other-race effect, Visual Cognition, 21 (2013), 1081-1095. doi: 10.1080/13506285.2013.811455. [17] P. J. Phillips, F. Jiang, A. Narvekar, J. Ayyad and A. J. O'Toole, An other-race effect for face recognition algorithms, ACM Transactions on Applied Perception (TAP), (2010), 1-12. doi: 10.6028/NIST.IR.7666. [18] J. A. Russell, Cross-cultural similarities and differences in affective processing and expression, in Emotions and Affect in Human Factors and Human-Computer Interaction, Elsevier, 2017, 123-141. doi: 10.1016/B978-0-12-801851-4.00004-5. [19] J. M. Susskind, D. H. Lee, A. Cusi, R. Feiman, W. Grabski and A. K. Anderson, Expressing fear enhances sensory acquisition, Nature Neuroscience, 11 (2008), 843-850. doi: 10.1038/nn.2138. [20] M. Wang, W. Deng, J. Hu, J. Peng, X. Tao and Y. Huang, Racial faces in-the-wild: Reducing racial bias by deep unsupervised domain adaptation, arXiv preprint, arXiv: 1812.00194. [21] M. Xue, X. Duan, J. Zhou, C. Wang, Y. Wang, Z. Li and W. Liu, A computational other-race-effect analysis for 3d facial expression recognition, Chinese Conference on Biometric Recognition, 2016, 483-493. doi: 10.1007/978-3-319-46654-5_53. [22] M. Xue, A. Mian, W. Liu and L. Li, Fully automatic 3d facial expression recognition using local depth features, in IEEE Winter Conference on Applications of Computer Vision, IEEE, 2014, 1096-1103. [23] X. Yan, T. J. Andrews, R. Jenkins and A. W. Young, Cross-cultural differences and similarities underlying other-race effects for facial identity and expression, Quarterly Journal of Experimental Psychology, 69 (2016), 1247-1254. doi: 10.1080/17470218.2016.1146312. [24] L. Yin, X. Wei, Y. Sun, J. Wang and M. J. Rosato, A 3d facial expression database for facial behavior research, in 7th International Conference on Automatic Face and Gesture Recognition (FGR06), IEEE, 2006, 211-216. [25] Q. Zhen, D. Huang, Y. Wang and L. Chen, Muscular movement model based automatic 3d facial expression recognition, in International Conference on Multimedia Modeling, Springer, 2015, 522-533. doi: 10.1007/978-3-319-14445-0_45. [26] Q. Zhen, D. Huang, Y. Wang and L. Chen, Muscular movement model-based automatic 3d/4d facial expression recognition, IEEE Transactions on Multimedia, 18 (2016), 1438-1450. doi: 10.1109/TMM.2016.2557063.
The average performance of facial expression recognition on East-Asian individuals when the ethnic-related features are removed gradually
The confusion matrix of multi-ethnic facial expression recognition before(a) and after(b) ethnic-related feature elimination based on East-Asian individuals
The average performance of facial expression recognition on White individuals when the ethnic-related features are removed gradually
The confusion matrix of multi-ethnic facial expression recognition before(a) and after(b) ethnic-related feature elimination based on White individuals
The ethnicity distribution of BU-3DFE database
 Ethnicity Sample Size Number of 3D Faces White 51 1224 East-Asian 24 576 Black 9 216 Hispanic-Latino 8 192 Indian 6 144 Middle-East Asian 2 48
 Ethnicity Sample Size Number of 3D Faces White 51 1224 East-Asian 24 576 Black 9 216 Hispanic-Latino 8 192 Indian 6 144 Middle-East Asian 2 48
 [1] Jian Zhao, Fang Deng, Jian Jia, Chunmeng Wu, Haibo Li, Yuan Shi, Shunli Zhang. A new face feature point matrix based on geometric features and illumination models for facial attraction analysis. Discrete & Continuous Dynamical Systems - S, 2019, 12 (4&5) : 1065-1072. doi: 10.3934/dcdss.2019073 [2] Qiang Yin, Gongfa Li, Jianguo Zhu. Research on the method of step feature extraction for EOD robot based on 2D laser radar. Discrete & Continuous Dynamical Systems - S, 2015, 8 (6) : 1415-1421. doi: 10.3934/dcdss.2015.8.1415 [3] Jianguo Dai, Wenxue Huang, Yuanyi Pan. A category-based probabilistic approach to feature selection. Big Data & Information Analytics, 2017, 2 (5) : 1-8. doi: 10.3934/bdia.2017020 [4] Zhi Guo Feng, K. F. Cedric Yiu, K.L. Mak. Feature extraction of the patterned textile with deformations via optimal control theory. Discrete & Continuous Dynamical Systems - B, 2011, 16 (4) : 1055-1069. doi: 10.3934/dcdsb.2011.16.1055 [5] Yunmei Lu, Mingyuan Yan, Meng Han, Qingliang Yang, Yanqing Zhang. Privacy preserving feature selection and Multiclass Classification for horizontally distributed data. Mathematical Foundations of Computing, 2018, 1 (4) : 331-348. doi: 10.3934/mfc.2018016 [6] Mohamed A. Tawhid, Kevin B. Dsouza. Hybrid binary dragonfly enhanced particle swarm optimization algorithm for solving feature selection problems. Mathematical Foundations of Computing, 2018, 1 (2) : 181-200. doi: 10.3934/mfc.2018009 [7] Makram Hamouda, Chang-Yeol Jung, Roger Temam. Asymptotic analysis for the 3D primitive equations in a channel. Discrete & Continuous Dynamical Systems - S, 2013, 6 (2) : 401-422. doi: 10.3934/dcdss.2013.6.401 [8] Yong Zhou. Remarks on regularities for the 3D MHD equations. Discrete & Continuous Dynamical Systems - A, 2005, 12 (5) : 881-886. doi: 10.3934/dcds.2005.12.881 [9] Hyeong-Ohk Bae, Bum Ja Jin. Estimates of the wake for the 3D Oseen equations. Discrete & Continuous Dynamical Systems - B, 2008, 10 (1) : 1-18. doi: 10.3934/dcdsb.2008.10.1 [10] Indranil SenGupta, Weisheng Jiang, Bo Sun, Maria Christina Mariani. Superradiance problem in a 3D annular domain. Conference Publications, 2011, 2011 (Special) : 1309-1318. doi: 10.3934/proc.2011.2011.1309 [11] Giovanny Guerrero, José Antonio Langa, Antonio Suárez. Biodiversity and vulnerability in a 3D mutualistic system. Discrete & Continuous Dynamical Systems - A, 2014, 34 (10) : 4107-4126. doi: 10.3934/dcds.2014.34.4107 [12] Sadek Gala. A new regularity criterion for the 3D MHD equations in $R^3$. Communications on Pure & Applied Analysis, 2012, 11 (3) : 973-980. doi: 10.3934/cpaa.2012.11.973 [13] Marko Filipović, Ivica Kopriva. A comparison of dictionary based approaches to inpainting and denoising with an emphasis to independent component analysis learned dictionaries. Inverse Problems & Imaging, 2011, 5 (4) : 815-841. doi: 10.3934/ipi.2011.5.815 [14] Danuta Gaweł, Krzysztof Fujarewicz. On the sensitivity of feature ranked lists for large-scale biological data. Mathematical Biosciences & Engineering, 2013, 10 (3) : 667-690. doi: 10.3934/mbe.2013.10.667 [15] Lok Ming Lui, Yalin Wang, Tony F. Chan, Paul M. Thompson. Brain anatomical feature detection by solving partial differential equations on general manifolds. Discrete & Continuous Dynamical Systems - B, 2007, 7 (3) : 605-618. doi: 10.3934/dcdsb.2007.7.605 [16] Jiahong Wu. Regularity results for weak solutions of the 3D MHD equations. Discrete & Continuous Dynamical Systems - A, 2004, 10 (1&2) : 543-556. doi: 10.3934/dcds.2004.10.543 [17] Gabriel Deugoue. Approximation of the trajectory attractor of the 3D MHD System. Communications on Pure & Applied Analysis, 2013, 12 (5) : 2119-2144. doi: 10.3934/cpaa.2013.12.2119 [18] Alp Eden, Varga K. Kalantarov. 3D convective Cahn--Hilliard equation. Communications on Pure & Applied Analysis, 2007, 6 (4) : 1075-1086. doi: 10.3934/cpaa.2007.6.1075 [19] Chongsheng Cao. Sufficient conditions for the regularity to the 3D Navier-Stokes equations. Discrete & Continuous Dynamical Systems - A, 2010, 26 (4) : 1141-1151. doi: 10.3934/dcds.2010.26.1141 [20] Ning Ju. The global attractor for the solutions to the 3D viscous primitive equations. Discrete & Continuous Dynamical Systems - A, 2007, 17 (1) : 159-179. doi: 10.3934/dcds.2007.17.159

Impact Factor:

## Tools

Article outline

Figures and Tables