# American Institute of Mathematical Sciences

August  2019, 13(4): 879-901. doi: 10.3934/ipi.2019040

## Two gesture-computing approaches by using electromagnetic waves

 1 Department of Mathematics, Harbin Institute of Technology, Harbin, 150001, China 2 Department of Mathematics, Southern University of Science and Technology, Shenzhen, 518055, China 3 Department of Mathematics, Hong Kong Baptist University, Kowloon, Hong Kong SAR, China

* Corresponding authors: Jingzhi Li and Hongyu Liu

Received  February 2019 Published  May 2019

We are concerned with a novel sensor-based gesture input/ instruction technology which enables human beings to interact with computers conveniently. The human being wears an emitter on the finger or holds a digital pen that generates a time harmonic point charge. The inputs/instructions are performed through moving the finger or the digital pen. The computer recognizes the instruction by determining the motion trajectory of the dynamic point charge from the collected electromagnetic field measurement data. The identification process is mathematically modelled as a dynamic inverse source problem for time-dependent Maxwell's equations. From a practical point of view, the point source should be assumed to move in an unknown inhomogeneous background medium, which models the human body and the surroundings. Moreover, a salient feature is that the electromagnetic radiated data are only collected in a limited aperture. For the inverse problem, we develop, from the respectively deterministic and stochastic viewpoints, a dynamic direct sampling method and a modified particle filter method. Both approaches can effectively recover the motion trajectory. Rigorous theoretical justifications are presented for the mathematical modelling and the proposed recovery methods. Extensive numerical experiments are conducted to illustrate the promising features of the two proposed recognition approaches.

Citation: Xianchao Wang, Yukun Guo, Jingzhi Li, Hongyu Liu. Two gesture-computing approaches by using electromagnetic waves. Inverse Problems & Imaging, 2019, 13 (4) : 879-901. doi: 10.3934/ipi.2019040
##### References:
 [1] R. Albanese and P. Monk, The inverse source problem for Maxwell's equations, Inverse Problems, 22 (2006), 1023-1035. doi: 10.1088/0266-5611/22/3/018. Google Scholar [2] H. Ammari, G. Bao and J. Fleming, An inverse source problem for Maxwell's equations in magnetoencephalography, SIAM J. Appl. Math., 62 (2002), 1369-1382. doi: 10.1137/S0036139900373927. Google Scholar [3] H. Ammari, T. Boulier, J. Garnier, H. Kang and H. Wang, Tracking of a mobile target using generalized polarization tensor, SIAM J. Imaging Sci., 6 (2013), 1477-1498. doi: 10.1137/120891022. Google Scholar [4] H. Ammari, M. P. Tran and H. Wang, Shape identification and classification in echolocation, SIAM J. Imaging Sci., 7 (2014), 1883-1905. doi: 10.1137/14096164X. Google Scholar [5] C. Andrieu, A. Doucet and E. Punskaya, Sequential Monte Carlo Methods in Practice, Springer-Verlag, New York, 2013.Google Scholar [6] M. Cheney and B. Borden, Imaging moving targets from scattered waves, Inverse Problems, 24 (2008), 035005, 22pp. doi: 10.1088/0266-5611/24/3/035005. Google Scholar [7] D. Colton and R. Kress, Inverse Acoustic and Electromagnetic Scattering Theory, 3$^{rd}$ Edition, Springer-Verlag, New York, 2013. doi: 10.1007/978-1-4614-4942-3. Google Scholar [8] D. Crisan and A. Doucet, A survey of convergence results on particle filtering methods for practitioners, IEEE Trans. Signal Process, 50 (2002), 736-746. doi: 10.1109/78.984773. Google Scholar [9] Y. Deng, H. Liu and G. Uhlmann, On an inverse boundary problem arising in brain imaging, Journal of Differential Equations, 2019, arXiv: 1702.00154 doi: 10.1016/j.jde.2019.03.019. Google Scholar [10] S. Escalera, V. Athitsos and I. Guyon, Challenges in multi-modal gesture recognition, Journal of Machine Learning Research, 17 (2016), Paper No. 72, 54 pp. Google Scholar [11] A. Fokas, Y. Kurylev and V. Marinakis, The unique determination of neuronal currents in the brain via magnetoencephalography, Inverse Problems, 20 (2004), 1067-1082. doi: 10.1088/0266-5611/20/4/005. Google Scholar [12] J. Garnier and M. Fink, Super-resolution in time-reversal focusing on a moving source, Wave Motion, 53 (2015), 80-93. doi: 10.1016/j.wavemoti.2014.11.005. Google Scholar [13] D. Griffiths, Introduction to Electrodynamics, 3$^{rd}$ Edition, Prentice Hall, New Jersey, 1999.Google Scholar [14] S. He and V. Romanov, Identification of dipole sources in a bounded domain for Maxwell's equations, Wave Motion, 28 (1998), 25-40. doi: 10.1016/S0165-2125(97)00063-2. Google Scholar [15] V. Isakov, Inverse Source Problems, Mathematical Surveys and Monographs, 34. American Mathematical Society, Providence, 1990. doi: 10.1090/surv/034. Google Scholar [16] J. Kaipio and E. Somersalo, Statistical and Computatioanal Inverse Problems, Springer-Verlag, New York, 2005. Google Scholar [17] M. V. Klibanov and V. G. Romanov, Two reconstruction procedures for a 3D phaseless inverse scattering problem for the generalized Helmholtz equation, Inverse Problems, 32 (2016), 015005, 16pp. doi: 10.1088/0266-5611/32/1/015005. Google Scholar [18] R. Leis, Initial Boundary Value Problems in Mathematical Physics, Springer Fachmedien, Wiesbaden, 1986. doi: 10.1007/978-3-663-10649-4. Google Scholar [19] J. Li, H. Liu, Z. Shang and H. Sun, Two single-shot methods for locating multiple electromagnetic scatterers, SIAM J. Appl. Math., 73 (2013), 1721-1746. doi: 10.1137/130907690. Google Scholar [20] Y. Liu, D. Jiang and M. Yamamoto, Inverse source problem for a double hyperbolic equation describing the three-dimensional time cone model, SIAM J. Appl. Math., 75 (2015), 2610-2635. doi: 10.1137/15M1018836. Google Scholar [21] H. Liu, Y. Wang and C. Yang, Mathematical design of a novel gesture-based instruction/input device using wave detection, SIAM J. Imaging Sci., 9 (2016), 822-841. doi: 10.1137/16M1063551. Google Scholar [22] S. Mitra and T. Acharya, Gesture Recognition: A Survey, IEEE Transactions on Systems Man and Cybernetics Part C, 37 (2007), 311-324. doi: 10.1109/TSMCC.2007.893280. Google Scholar [23] E. Nakaguchi, H. Inui and K. Ohnaka, An algebraic reconstruction of a moving point source for a scalar wave equation, Inverse Problems, 28 (2012), 065018, 21pp. doi: 10.1088/0266-5611/28/6/065018. Google Scholar [24] G. Nakamura and R. Potthast, Inverse Modeling, IOP Publishing, Bristol, 2015. Google Scholar [25] J. Nédélec, Acoustic and Electromagnetic Equations: Integral Representations for Harmonic Problems, Springer-Verlag, New York, 2001. doi: 10.1007/978-1-4757-4393-7. Google Scholar [26] X. Wang, Y. Guo, J. Li and H. Liu, Mathematical design of a novel input/instruction device using a moving emitter, Inverse Problems, 33 (2017), 105009, 19pp. doi: 10.1088/1361-6420/aa873f. Google Scholar [27] [28] S. Yang, P. K. Premaratne and P. J. Vial, Hand gesture recognition: An overview, in 5th IEEE International Conference on Broadband Network and Multimedia Technology, Academic Press, (2013), 63–69. doi: 10.1109/ICBNMT.2013.6823916. Google Scholar

show all references

##### References:
 [1] R. Albanese and P. Monk, The inverse source problem for Maxwell's equations, Inverse Problems, 22 (2006), 1023-1035. doi: 10.1088/0266-5611/22/3/018. Google Scholar [2] H. Ammari, G. Bao and J. Fleming, An inverse source problem for Maxwell's equations in magnetoencephalography, SIAM J. Appl. Math., 62 (2002), 1369-1382. doi: 10.1137/S0036139900373927. Google Scholar [3] H. Ammari, T. Boulier, J. Garnier, H. Kang and H. Wang, Tracking of a mobile target using generalized polarization tensor, SIAM J. Imaging Sci., 6 (2013), 1477-1498. doi: 10.1137/120891022. Google Scholar [4] H. Ammari, M. P. Tran and H. Wang, Shape identification and classification in echolocation, SIAM J. Imaging Sci., 7 (2014), 1883-1905. doi: 10.1137/14096164X. Google Scholar [5] C. Andrieu, A. Doucet and E. Punskaya, Sequential Monte Carlo Methods in Practice, Springer-Verlag, New York, 2013.Google Scholar [6] M. Cheney and B. Borden, Imaging moving targets from scattered waves, Inverse Problems, 24 (2008), 035005, 22pp. doi: 10.1088/0266-5611/24/3/035005. Google Scholar [7] D. Colton and R. Kress, Inverse Acoustic and Electromagnetic Scattering Theory, 3$^{rd}$ Edition, Springer-Verlag, New York, 2013. doi: 10.1007/978-1-4614-4942-3. Google Scholar [8] D. Crisan and A. Doucet, A survey of convergence results on particle filtering methods for practitioners, IEEE Trans. Signal Process, 50 (2002), 736-746. doi: 10.1109/78.984773. Google Scholar [9] Y. Deng, H. Liu and G. Uhlmann, On an inverse boundary problem arising in brain imaging, Journal of Differential Equations, 2019, arXiv: 1702.00154 doi: 10.1016/j.jde.2019.03.019. Google Scholar [10] S. Escalera, V. Athitsos and I. Guyon, Challenges in multi-modal gesture recognition, Journal of Machine Learning Research, 17 (2016), Paper No. 72, 54 pp. Google Scholar [11] A. Fokas, Y. Kurylev and V. Marinakis, The unique determination of neuronal currents in the brain via magnetoencephalography, Inverse Problems, 20 (2004), 1067-1082. doi: 10.1088/0266-5611/20/4/005. Google Scholar [12] J. Garnier and M. Fink, Super-resolution in time-reversal focusing on a moving source, Wave Motion, 53 (2015), 80-93. doi: 10.1016/j.wavemoti.2014.11.005. Google Scholar [13] D. Griffiths, Introduction to Electrodynamics, 3$^{rd}$ Edition, Prentice Hall, New Jersey, 1999.Google Scholar [14] S. He and V. Romanov, Identification of dipole sources in a bounded domain for Maxwell's equations, Wave Motion, 28 (1998), 25-40. doi: 10.1016/S0165-2125(97)00063-2. Google Scholar [15] V. Isakov, Inverse Source Problems, Mathematical Surveys and Monographs, 34. American Mathematical Society, Providence, 1990. doi: 10.1090/surv/034. Google Scholar [16] J. Kaipio and E. Somersalo, Statistical and Computatioanal Inverse Problems, Springer-Verlag, New York, 2005. Google Scholar [17] M. V. Klibanov and V. G. Romanov, Two reconstruction procedures for a 3D phaseless inverse scattering problem for the generalized Helmholtz equation, Inverse Problems, 32 (2016), 015005, 16pp. doi: 10.1088/0266-5611/32/1/015005. Google Scholar [18] R. Leis, Initial Boundary Value Problems in Mathematical Physics, Springer Fachmedien, Wiesbaden, 1986. doi: 10.1007/978-3-663-10649-4. Google Scholar [19] J. Li, H. Liu, Z. Shang and H. Sun, Two single-shot methods for locating multiple electromagnetic scatterers, SIAM J. Appl. Math., 73 (2013), 1721-1746. doi: 10.1137/130907690. Google Scholar [20] Y. Liu, D. Jiang and M. Yamamoto, Inverse source problem for a double hyperbolic equation describing the three-dimensional time cone model, SIAM J. Appl. Math., 75 (2015), 2610-2635. doi: 10.1137/15M1018836. Google Scholar [21] H. Liu, Y. Wang and C. Yang, Mathematical design of a novel gesture-based instruction/input device using wave detection, SIAM J. Imaging Sci., 9 (2016), 822-841. doi: 10.1137/16M1063551. Google Scholar [22] S. Mitra and T. Acharya, Gesture Recognition: A Survey, IEEE Transactions on Systems Man and Cybernetics Part C, 37 (2007), 311-324. doi: 10.1109/TSMCC.2007.893280. Google Scholar [23] E. Nakaguchi, H. Inui and K. Ohnaka, An algebraic reconstruction of a moving point source for a scalar wave equation, Inverse Problems, 28 (2012), 065018, 21pp. doi: 10.1088/0266-5611/28/6/065018. Google Scholar [24] G. Nakamura and R. Potthast, Inverse Modeling, IOP Publishing, Bristol, 2015. Google Scholar [25] J. Nédélec, Acoustic and Electromagnetic Equations: Integral Representations for Harmonic Problems, Springer-Verlag, New York, 2001. doi: 10.1007/978-1-4757-4393-7. Google Scholar [26] X. Wang, Y. Guo, J. Li and H. Liu, Mathematical design of a novel input/instruction device using a moving emitter, Inverse Problems, 33 (2017), 105009, 19pp. doi: 10.1088/1361-6420/aa873f. Google Scholar [27] [28] S. Yang, P. K. Premaratne and P. J. Vial, Hand gesture recognition: An overview, in 5th IEEE International Conference on Broadband Network and Multimedia Technology, Academic Press, (2013), 63–69. doi: 10.1109/ICBNMT.2013.6823916. Google Scholar
Schematic illustration of the proposed input/ instruction technology using a moving emitter
The measurement points and human being's motion domain
Reconstruction of the trajectory "C". (a) exact moving trajectory in the homogeneous medium, (b) exact moving trajectory in the inhomogeneous medium, (c), (d), (e), the electric field in the three components with the receiver at $(2,2,1)$, where the red solid line denotes the homogeneous medium and the blue dotted line denotes the inhomogeneous medium, (f) the relationship between the $\alpha$ and the error, (g) reconstruction by the particle filter method in the homogeneous medium, (h) reconstruction by the particle filter method in the inhomogeneous medium
Reconstruction of the trajectory "3". (a) exact moving trajectory, (b) reconstruction by the direct sampling method, (c) plots of Indicator function $I(\mathit{\boldsymbol{y}},t)$ in the instant $t = 4.5$ s (slices at $x = 1$ and $y = 1.2$), (d) plots of Indicator function $I(\mathit{\boldsymbol{y}},t)$ in the instant $t = 7.5$ s (slices at $x = 1$ and $y = 1.2$), (e) reconstruction by the particle filter method with $N_s = 100$, (f) point-wise error between the exact and reconstructed trajectory
Reconstruction of a conical spiral shaped trajectory. (a) exact moving trajectory, (b) reconstruction result by the direct sampling method, (c) reconstruction by the particle filter method with $N_s = 200$, (d) point-wise error between the exact and reconstructed trajectory, (e) relationship between the number of particles and relative $L^2$ error
Reconstruct the trajectory of a text. (a) exact moving trajectory, (b) reconstruction by the direct sampling method, (c) reconstruction by the particle filter method with $N_s = 500$, (d) point-wise error between the exact and reconstructed trajectory
Reconstruct the moving trajectory by the particle filter method and snapshots at different instants, where the black points denotes the particles. (a) $t = 3.0$ s, (b) $t = 6.6$ s, (c) $t = 9.0$ s, (d) $t = 9.1$ s, (e) $t = 9.2$ s, (f) $t = 12.0$ s
The relative $L^2$ error and the CPU time for reconstructing the trajectory "3" ($N_h$ and $N_s$ denote, respectively, the number of sampling points and the number of particle samples)
 Direct sampling method Particle filter technique $N_h=25^3$ $N_h=50^3$ $N_h=100^3$ $N_s=50$ $N_s=100$ $N_s=500$ Error 2.09% 0.79% 0.68% 6.76% 2.62% 1.59% Time 16 s 115 s 897 s 0.23 s 0.48 s 10.5 s
 Direct sampling method Particle filter technique $N_h=25^3$ $N_h=50^3$ $N_h=100^3$ $N_s=50$ $N_s=100$ $N_s=500$ Error 2.09% 0.79% 0.68% 6.76% 2.62% 1.59% Time 16 s 115 s 897 s 0.23 s 0.48 s 10.5 s
 [1] Qifeng Cheng, Xue Han, Tingting Zhao, V S Sarma Yadavalli. Improved particle swarm optimization and neighborhood field optimization by introducing the re-sampling step of particle filter. Journal of Industrial & Management Optimization, 2019, 15 (1) : 177-198. doi: 10.3934/jimo.2018038 [2] Jingzhi Li, Jun Zou. A direct sampling method for inverse scattering using far-field data. Inverse Problems & Imaging, 2013, 7 (3) : 757-775. doi: 10.3934/ipi.2013.7.757 [3] Frederic Weidling, Thorsten Hohage. Variational source conditions and stability estimates for inverse electromagnetic medium scattering problems. Inverse Problems & Imaging, 2017, 11 (1) : 203-220. doi: 10.3934/ipi.2017010 [4] Peter Monk, Virginia Selgas. Sampling type methods for an inverse waveguide problem. Inverse Problems & Imaging, 2012, 6 (4) : 709-747. doi: 10.3934/ipi.2012.6.709 [5] Jingzhi Li, Hongyu Liu, Hongpeng Sun. On a gesture-computing technique using electromagnetic waves. Inverse Problems & Imaging, 2018, 12 (3) : 677-696. doi: 10.3934/ipi.2018029 [6] Alexandr Mikhaylov, Victor Mikhaylov. Dynamic inverse problem for Jacobi matrices. Inverse Problems & Imaging, 2019, 13 (3) : 431-447. doi: 10.3934/ipi.2019021 [7] Christodoulos E. Athanasiadis, Vassilios Sevroglou, Konstantinos I. Skourogiannis. The inverse electromagnetic scattering problem by a mixed impedance screen in chiral media. Inverse Problems & Imaging, 2015, 9 (4) : 951-970. doi: 10.3934/ipi.2015.9.951 [8] Masaru Ikehata. The enclosure method for inverse obstacle scattering using a single electromagnetic wave in time domain. Inverse Problems & Imaging, 2016, 10 (1) : 131-163. doi: 10.3934/ipi.2016.10.131 [9] Peijun Li, Ganghua Yuan. Increasing stability for the inverse source scattering problem with multi-frequencies. Inverse Problems & Imaging, 2017, 11 (4) : 745-759. doi: 10.3934/ipi.2017035 [10] Kenichi Sakamoto, Masahiro Yamamoto. Inverse source problem with a final overdetermination for a fractional diffusion equation. Mathematical Control & Related Fields, 2011, 1 (4) : 509-518. doi: 10.3934/mcrf.2011.1.509 [11] Shumin Li, Masahiro Yamamoto, Bernadette Miara. A Carleman estimate for the linear shallow shell equation and an inverse source problem. Discrete & Continuous Dynamical Systems - A, 2009, 23 (1&2) : 367-380. doi: 10.3934/dcds.2009.23.367 [12] Yuxuan Gong, Xiang Xu. Inverse random source problem for biharmonic equation in two dimensions. Inverse Problems & Imaging, 2019, 13 (3) : 635-652. doi: 10.3934/ipi.2019029 [13] Peter Monk, Virginia Selgas. Near field sampling type methods for the inverse fluid--solid interaction problem. Inverse Problems & Imaging, 2011, 5 (2) : 465-483. doi: 10.3934/ipi.2011.5.465 [14] Xiaoying Han, Jinglai Li, Dongbin Xiu. Error analysis for numerical formulation of particle filter. Discrete & Continuous Dynamical Systems - B, 2015, 20 (5) : 1337-1354. doi: 10.3934/dcdsb.2015.20.1337 [15] Jingzhi Li, Hongyu Liu, Qi Wang. Fast imaging of electromagnetic scatterers by a two-stage multilevel sampling method. Discrete & Continuous Dynamical Systems - S, 2015, 8 (3) : 547-561. doi: 10.3934/dcdss.2015.8.547 [16] Gen Nakamura, Michiyuki Watanabe. An inverse boundary value problem for a nonlinear wave equation. Inverse Problems & Imaging, 2008, 2 (1) : 121-131. doi: 10.3934/ipi.2008.2.121 [17] Jun Lai, Ming Li, Peijun Li, Wei Li. A fast direct imaging method for the inverse obstacle scattering problem with nonlinear point scatterers. Inverse Problems & Imaging, 2018, 12 (3) : 635-665. doi: 10.3934/ipi.2018027 [18] Zhiming Chen, Shaofeng Fang, Guanghui Huang. A direct imaging method for the half-space inverse scattering problem with phaseless data. Inverse Problems & Imaging, 2017, 11 (5) : 901-916. doi: 10.3934/ipi.2017042 [19] Guanghui Hu, Peijun Li, Xiaodong Liu, Yue Zhao. Inverse source problems in electrodynamics. Inverse Problems & Imaging, 2018, 12 (6) : 1411-1428. doi: 10.3934/ipi.2018059 [20] Junfeng Yang. Dynamic power price problem: An inverse variational inequality approach. Journal of Industrial & Management Optimization, 2008, 4 (4) : 673-684. doi: 10.3934/jimo.2008.4.673

2018 Impact Factor: 1.469

## Tools

Article outline

Figures and Tables