• Previous Article
    Variational source conditions and stability estimates for inverse electromagnetic medium scattering problems
  • IPI Home
  • This Issue
  • Next Article
    Non-linear Tikhonov regularization in Banach spaces for inverse scattering from anisotropic penetrable media
January  2017, 11(1): 177-202. doi: 10.3934/ipi.2017009

Foveated compressive imaging for low power vehicle fingerprinting and tracking in aerial imagery

HRL Laboratories LLC, 3011 Malibu Canyon Road, Malibu, CA 90265-4797, USA

Received  January 2015 Revised  September 2016 Published  January 2017

Fund Project: This work is supported by the Defense Advanced Research Projects Agency and SPAWAR Systems Center Pacific under Contract No. N66001-11-C-4001

We describe a foveated compressive sensing approach for image analysis applications that utilizes knowledge of the task to be performed to reduce the number of required sensor measurements and sensor size, weight, and power (SWAP) compared to conventional Nyquist sampling and compressive sensing-based approaches. Our Compressive Optical Foveated Architecture (COFA) adapts the dictionary and compressive measurements to structure and sparsity in the signal, task, and scene by reducing measurement and dictionary mutual coherence and increasing sparsity using principles of actionable information and foveated compressive sensing. Actionable information is used to extract task-relevant regions of interest (ROIs) from a low-resolution scene analysis by eliminating the effects of nuisances for occlusion and anomalous motion detection. From the extracted ROIs, preferential measurements are taken using foveation as part of the compressive sensing adaptation process. The task-specific measurement matrix is optimized by using a novel saliency-weighted coherence minimization with respect to the learned signal dictionary. This incorporates the relative usage of the atoms in the dictionary. We utilize a patch-based method to learn the signal priors. A tree-structured dictionary of image patches using K-SVD is learned which can sparsely represent any given image patch with the tree structure. We have implemented COFA in an end-to-end simulation of a vehicle fingerprinting task for aerial surveillance using foveated compressive measurements adapted to hierarchical ROIs consisting of background, roads, and vehicles. Our results show 113× reduction in measurements over conventional sensing and 28× reduction over compressive sensing using random measurements.

Citation: Kang-Yu Ni, Shankar Rao, Yuri Owechko. Foveated compressive imaging for low power vehicle fingerprinting and tracking in aerial imagery. Inverse Problems & Imaging, 2017, 11 (1) : 177-202. doi: 10.3934/ipi.2017009
References:
[1]

M. AharonM. Elad and A. Bruckstein, K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation, IEEE Transactions on Signal Processing, 54 (2006), 4311-4322. Google Scholar

[2]

A. Ayvaci, M. Raptis and S. Soatto, Occlusion Detection and Motion Estimation with Convex Optimization Neural Information Processing Systems, 2010.Google Scholar

[3]

A. BrucksteinD. Donoho and M. Elad, From sparse solutions of systems of equations to sparse modeling of signals and images, SIAM Review, 51 (2009), 34-81. doi: 10.1137/060657704. Google Scholar

[4]

E. Candés and T. Tao, Decoding by linear programming, IEEE Trans. Inform. Theory, 51 (2005), 4203-4215. doi: 10.1109/TIT.2005.858979. Google Scholar

[5]

I. Ciocoiu, Foveated compressed sensing, Proc. of Europe. Conf. on Circuit Theory and Design, (2011), 29-32. doi: 10.1109/ECCTD.2011.6043336. Google Scholar

[6]

Columbus surrogate unmanned aerial vehicle (CSUAV) dataset, United States Air Force Research Lab (AFRL).Google Scholar

[7]

J. P. CurzanC. R. Baxter and M. A. Massie, Variable acuity imager with dynamically steerable, programmable superpixels, Infrared Technology and Applications, Proc. SPIE, 4820 (2003), p318. doi: 10.1117/12.451183. Google Scholar

[8]

D. DonohoA. Maleki and A. Montanari, Noise sensitivity phase transition in compressed sensing, IEEE Transactions on Information Theory, 57 (2011), 6920-6941. doi: 10.1109/TIT.2011.2165823. Google Scholar

[9]

J. Duarte-Carvajalino and G. Sapiro, Learning to sense sparse signals: Simultaneous sensing matrix and sparsifying dictionary optimization, IEEE Transactions on Image Processing, 18 (2009), 1395-1408. doi: 10.1109/TIP.2009.2022459. Google Scholar

[10]

G. Georgiadis, A. Ayvaci and S. Soatto, Actionable Saliency Detection Proc. of CVPR, 2012.Google Scholar

[11]

Z. HarmanyA. OhR. Marcia and R. Willet, Motion-adaptive compressive coded apertures, Proc. of SPIE, 8165 (2011), 1-5. doi: 10.1117/12.892726. Google Scholar

[12]

D. Heeger and A. Jepson, Subspace methods for recovering rigid motion, Intl. J. of Comp. Vis., 7 (1992), 95-117. Google Scholar

[13]

InView Shortwave Infrared (SWIR) Cameras, http://inviewcorp.com/products/shortwave-infrared-swir-cameras/.Google Scholar

[14]

R. Jenatton, J. Mairal, G. Obozinski and F. Bach, Proximal Methods for Sparse Hierarchical Dictionary Learning J. Machine Learning Research, 2011.Google Scholar

[15]

R. Larcom and T. Coffman, Foveated image formation through compressive sensing, Proc. of Southwest Symp. Image Anal. Interp., (2010), 145-148. doi: 10.1109/SSIAI.2010.5483896. Google Scholar

[16]

T. MundhenkK. NiK. Kim and Y. Owechko, Detection of unknown targets from aerial camera and extraction of simple object fingerprints for the purpose of target reacquisition, Proc. of SPIE, 8301 (2012), 1-14. doi: 10.1117/12.906491. Google Scholar

[17]

S. Soatto, Steps Towards a Theory of Visual Information Textbook Draft.Google Scholar

[18]

A. Soni and J. Haupt, Efficient adaptive compressive sensing using sparse hierarchical learned dictionaries, Proc. of ASILOMAR, (2011), 1250-1254. doi: 10.1109/ACSSC.2011.6190216. Google Scholar

[19]

P. D. Sturkie, Sturkie's Avian Physiology 5th Edition, Academic Press, San Diego.Google Scholar

[20]

N. Sundaram, T. Brox and K. Keutzer, Dense point trajectories by GPU-accelerated large displacement optical flow, Chapter: Computer Vision C ECCV 2010, Volume 6311 of the series Lecture Notes in Computer Science, (2010), 438-451. doi: 10.1007/978-3-642-15549-9_32. Google Scholar

[21]

F. Tanner, B. Colder, C. Pullen, D. Heagy, M. Eppolito, V. Carlan, C. Oertel and P. Sallee, Overhead Imagery Research Data Set: An Annotated Data Library and Tools to aid in the Development of Computer Vision Algorithms Proc. of IEEE Applied Imagery Pattern Rec. Workshop, 2009. doi: 10.1109/AIPR.2009.5466304. Google Scholar

[22]

L. Zelnik-ManorK. Rosenblum and Y. Eldar, Sensing matrix optimization for block-sparse decoding, IEEE Transactions on Signal Processing, 59 (2011), 4300-4312. doi: 10.1109/TSP.2011.2159211. Google Scholar

show all references

References:
[1]

M. AharonM. Elad and A. Bruckstein, K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation, IEEE Transactions on Signal Processing, 54 (2006), 4311-4322. Google Scholar

[2]

A. Ayvaci, M. Raptis and S. Soatto, Occlusion Detection and Motion Estimation with Convex Optimization Neural Information Processing Systems, 2010.Google Scholar

[3]

A. BrucksteinD. Donoho and M. Elad, From sparse solutions of systems of equations to sparse modeling of signals and images, SIAM Review, 51 (2009), 34-81. doi: 10.1137/060657704. Google Scholar

[4]

E. Candés and T. Tao, Decoding by linear programming, IEEE Trans. Inform. Theory, 51 (2005), 4203-4215. doi: 10.1109/TIT.2005.858979. Google Scholar

[5]

I. Ciocoiu, Foveated compressed sensing, Proc. of Europe. Conf. on Circuit Theory and Design, (2011), 29-32. doi: 10.1109/ECCTD.2011.6043336. Google Scholar

[6]

Columbus surrogate unmanned aerial vehicle (CSUAV) dataset, United States Air Force Research Lab (AFRL).Google Scholar

[7]

J. P. CurzanC. R. Baxter and M. A. Massie, Variable acuity imager with dynamically steerable, programmable superpixels, Infrared Technology and Applications, Proc. SPIE, 4820 (2003), p318. doi: 10.1117/12.451183. Google Scholar

[8]

D. DonohoA. Maleki and A. Montanari, Noise sensitivity phase transition in compressed sensing, IEEE Transactions on Information Theory, 57 (2011), 6920-6941. doi: 10.1109/TIT.2011.2165823. Google Scholar

[9]

J. Duarte-Carvajalino and G. Sapiro, Learning to sense sparse signals: Simultaneous sensing matrix and sparsifying dictionary optimization, IEEE Transactions on Image Processing, 18 (2009), 1395-1408. doi: 10.1109/TIP.2009.2022459. Google Scholar

[10]

G. Georgiadis, A. Ayvaci and S. Soatto, Actionable Saliency Detection Proc. of CVPR, 2012.Google Scholar

[11]

Z. HarmanyA. OhR. Marcia and R. Willet, Motion-adaptive compressive coded apertures, Proc. of SPIE, 8165 (2011), 1-5. doi: 10.1117/12.892726. Google Scholar

[12]

D. Heeger and A. Jepson, Subspace methods for recovering rigid motion, Intl. J. of Comp. Vis., 7 (1992), 95-117. Google Scholar

[13]

InView Shortwave Infrared (SWIR) Cameras, http://inviewcorp.com/products/shortwave-infrared-swir-cameras/.Google Scholar

[14]

R. Jenatton, J. Mairal, G. Obozinski and F. Bach, Proximal Methods for Sparse Hierarchical Dictionary Learning J. Machine Learning Research, 2011.Google Scholar

[15]

R. Larcom and T. Coffman, Foveated image formation through compressive sensing, Proc. of Southwest Symp. Image Anal. Interp., (2010), 145-148. doi: 10.1109/SSIAI.2010.5483896. Google Scholar

[16]

T. MundhenkK. NiK. Kim and Y. Owechko, Detection of unknown targets from aerial camera and extraction of simple object fingerprints for the purpose of target reacquisition, Proc. of SPIE, 8301 (2012), 1-14. doi: 10.1117/12.906491. Google Scholar

[17]

S. Soatto, Steps Towards a Theory of Visual Information Textbook Draft.Google Scholar

[18]

A. Soni and J. Haupt, Efficient adaptive compressive sensing using sparse hierarchical learned dictionaries, Proc. of ASILOMAR, (2011), 1250-1254. doi: 10.1109/ACSSC.2011.6190216. Google Scholar

[19]

P. D. Sturkie, Sturkie's Avian Physiology 5th Edition, Academic Press, San Diego.Google Scholar

[20]

N. Sundaram, T. Brox and K. Keutzer, Dense point trajectories by GPU-accelerated large displacement optical flow, Chapter: Computer Vision C ECCV 2010, Volume 6311 of the series Lecture Notes in Computer Science, (2010), 438-451. doi: 10.1007/978-3-642-15549-9_32. Google Scholar

[21]

F. Tanner, B. Colder, C. Pullen, D. Heagy, M. Eppolito, V. Carlan, C. Oertel and P. Sallee, Overhead Imagery Research Data Set: An Annotated Data Library and Tools to aid in the Development of Computer Vision Algorithms Proc. of IEEE Applied Imagery Pattern Rec. Workshop, 2009. doi: 10.1109/AIPR.2009.5466304. Google Scholar

[22]

L. Zelnik-ManorK. Rosenblum and Y. Eldar, Sensing matrix optimization for block-sparse decoding, IEEE Transactions on Signal Processing, 59 (2011), 4300-4312. doi: 10.1109/TSP.2011.2159211. Google Scholar

Figure 1.  (Top) Flowchart of overall approach for adaptation of foveated measurements and signal representations. (Bottom) Details of online scene-adaptive reconstruction for the vehicle fingerprinting task. Left: Dynamic input scene. Middle: Reconstructed low-resolution background scene that is used to detect ROIs using anomalous motion detection. Right: Reconstructed high resolution ROIs using adapted dictionary overlaid with low-resolution background. This representation reduces the total number of measurements $M = M_{\rm Backg}+ M_{\rm ROI}$ needed for the task.
Figure 2.  The Manx Shearwater seabird [19] has multiple hierarchical levels of fovea (Right) for acquisition and tracking.
Figure 3.  Our foveated compressive sensing optical architecture generates a composite image frame consisting of low-resolution background contextual information and the high-resolution task-relevant regions of interest (ROIs). A fixed budget of $M$ measurements can be adaptively divided between the background and ROIs, allowing background resolution to be traded for higher resolution ROIs. By adapting both the measurement matrix and the dictionary to the ROIs, the number of measurements needed for a given level of task performance can be greatly reduced
Figure 4.  Simulation results comparing conventional imaging, conventional CS imaging, and foveated CS imaging. The conventional CS imaging reconstructs the image from random DCT measurements via $\ell_1$-minimization. Imaging results were all obtained using 3025 measurements of the scene but foveated compressive sensing achieved much higher effective resolution in the region of interest (ROI) than conventional imaging while also reconstructing the context around the ROI
Figure  .  Algorithm 1: Iterative reweighted subspace minimization algorithm that we use to find salient regions in images.
Figure 5.  Detection of moving vehicle ROIs in two frames using Actionable Saliency despite camera motions. [10]
Figure 6.  Detection of moving vehicle ROIs from images reconstructed from different numbers of compressive sensing measurements.
Figure 7.  Top: Example images of cars from OIRDS [18] aerial views used for training our dictionary. Bottom left: Learned tree-structured dictionary with example atoms from each level of the tree. Bottom right: Distribution of coefficients over the training set
Figure 8.  Patch-based compressive optical foveated architecture (COFA) optical system
Figure 9.  Hierarchical layered regions of interest (ROIs) for the vehicle tracking and fingerprinting task. Layer 1 is the background, Layer 2 is the road, and Layer 3 contains the moving vehicles on the road
Figure 10.  Contours of the minimax noise sensitivity $M^*(\delta,\rho)$ in the $(\delta,\rho)$ plane. $\delta=M/N$ is the subsampling rate and $\rho=K/M$ is the sparsity. The dotted black curve graphs the phase boundary $M^*(\delta,\rho{\rm MSE}(\delta))$. Above this curve, $M^*(\delta,\rho)=\infty$. The colored lines represent level sets of $M*(\delta,\rho)$. (From [8])
Figure 11.  Reduction in measurements needed over conventional compressive sensing as function of ROI resolution and size for 2 and 3 ROI layers
Figure 12.  Reconstruction SNR for CSUAV scenes with 1$\times$, 2$\times$, and 4$\times$ downsampling
Figure 13.  Example reconstructed CSUAV frames using the wmc + tree method. Sufficient resolution is maintained with $25\%$ of ($160\times 120$) measurements or $1/64$ of the number of Nyquist samples to detect ROIs corresponding to moving vehicles in the scene
Figure 14.  Left, Middle: Reconstruction SNR for vehicles displayed graphically and numerically. Right: Example reconstructed vehicles vs. number of measurements and measurement/dictionary types
Figure 15.  Fingerprinting task performance results for reconstructed vehicle windows from CSUAV motion imagery. Baseline performance on original input windows is 76.17$\%$
Figure 16.  COFA simulation framework for vehicle fingerprinting. For simplicity, Layer 2 (road ROIs) is not shown
Figure 17.  Left: 3-layer ROI hierarchy for COFA pipeline. Right: Multi-resolution composite reconstruction of a CSUAV video frame. Note the variable resolution in the patches corresponding to different ROI types. The Car ROIs have the highest resolution
Figure 18.  Reconstruction SNR and noise sensitivity for CSUAV Layer 1 (Background). The results are averaged over all $16\times 16$ patches in 50 frames of CSUAV-11 video. Non-random measurements and structured dictionary resulted in 4$\times$ fewer measurements for the same SNR compared to random measurements. Left: Reconstruction SNR (dB) vs. measurements percentage $(M_{\rm ROI}/N_{\rm ROI})$. Right: Reconstruction SNR (dB) vs. added measurement noise level ($\%$) with fixed $6.25\%$ of measurements of Layer 1
Figure 19.  Reconstruction SNR and noise sensitivity for CSUAV Layer 2 (Road). The results are averaged over all $16\times 16$ patches in 50 frames of CSUAV-11 video. Non-random measurements and structured dictionary resulted in $>8\times$ fewer measurements for the same SNR compared to random measurements. Left: Reconstruction SNR (dB) vs. measurements percentage $(M_{\rm ROI}/N_{\rm ROI})$. Right: Reconstruction SNR (dB) vs. added measurement noise level ($\%$) with fixed $6.25\%$ of measurements of Layer 2
Figure 20.  Reconstruction SNR and noise sensitivity for CSUAV Layer 3 (Cars). The results are averaged over all $16\times 16$ patches in 50 frames of CSUAV-11 video. Non-random measurements and structured dictionary resulted in $4\times$ fewer measurements for the same SNR compared to random measurements. Left: Reconstruction SNR (dB) vs. measurements percentage $(M_{\rm ROI}/N_{\rm ROI})$. Right: Reconstruction SNR (dB) vs. added measurement noise level ($\%$) with fixed $25\%$ measurements of Layer 3
Figure 21.  Vehicle fingerprinting performance and noise sensitivity results for 3-layer pipeline. Left: Correct identification vs. measurements percentage $(M_{\rm ROI}/N_{\rm ROI})$. Right: Correct identification vs. added measurement noise level ($\%$) with fixed $25\%$ measurements of Layer 3
Table 1.  Tested methods and reconstruction algorithms
MethodMeasurementDictionary
rand + flatrandom Gaussian orthonormal measurements(flat) ksvd dictionary
rand + treerandom Gaussian orthonormal measurementshierarchical (tree) dictionary
mc + flatminimum coherence measurements(flat) ksvd dictionary
mc + treeminimum coherence measurementshierarchical (tree) dictionary
wmc + treeweighted minimum coherence measurementshierarchical (tree) dictionary
MethodMeasurementDictionary
rand + flatrandom Gaussian orthonormal measurements(flat) ksvd dictionary
rand + treerandom Gaussian orthonormal measurementshierarchical (tree) dictionary
mc + flatminimum coherence measurements(flat) ksvd dictionary
mc + treeminimum coherence measurementshierarchical (tree) dictionary
wmc + treeweighted minimum coherence measurementshierarchical (tree) dictionary
[1]

Yangyang Xu, Wotao Yin, Stanley Osher. Learning circulant sensing kernels. Inverse Problems & Imaging, 2014, 8 (3) : 901-923. doi: 10.3934/ipi.2014.8.901

[2]

Vikram Krishnamurthy, William Hoiles. Information diffusion in social sensing. Numerical Algebra, Control & Optimization, 2016, 6 (3) : 365-411. doi: 10.3934/naco.2016017

[3]

Jian-Wu Xue, Xiao-Kun Xu, Feng Zhang. Big data dynamic compressive sensing system architecture and optimization algorithm for internet of things. Discrete & Continuous Dynamical Systems - S, 2015, 8 (6) : 1401-1414. doi: 10.3934/dcdss.2015.8.1401

[4]

Hong Jiang, Wei Deng, Zuowei Shen. Surveillance video processing using compressive sensing. Inverse Problems & Imaging, 2012, 6 (2) : 201-214. doi: 10.3934/ipi.2012.6.201

[5]

Zhihua Zhang, Naoki Saito. PHLST with adaptive tiling and its application to antarctic remote sensing image approximation. Inverse Problems & Imaging, 2014, 8 (1) : 321-337. doi: 10.3934/ipi.2014.8.321

[6]

Yonggui Zhu, Yuying Shi, Bin Zhang, Xinyan Yu. Weighted-average alternating minimization method for magnetic resonance image reconstruction based on compressive sensing. Inverse Problems & Imaging, 2014, 8 (3) : 925-937. doi: 10.3934/ipi.2014.8.925

[7]

Yingying Li, Stanley Osher. Coordinate descent optimization for l1 minimization with application to compressed sensing; a greedy algorithm. Inverse Problems & Imaging, 2009, 3 (3) : 487-503. doi: 10.3934/ipi.2009.3.487

[8]

Jae Deok Kim, Ganguk Hwang. Cross-layer modeling and optimization of multi-channel cognitive radio networks under imperfect channel sensing. Journal of Industrial & Management Optimization, 2015, 11 (3) : 807-828. doi: 10.3934/jimo.2015.11.807

[9]

Steven L. Brunton, Joshua L. Proctor, Jonathan H. Tu, J. Nathan Kutz. Compressed sensing and dynamic mode decomposition. Journal of Computational Dynamics, 2015, 2 (2) : 165-191. doi: 10.3934/jcd.2015002

[10]

Ying Zhang, Ling Ma, Zheng-Hai Huang. On phaseless compressed sensing with partially known support. Journal of Industrial & Management Optimization, 2017, 13 (5) : 1-8. doi: 10.3934/jimo.2019014

[11]

Cesare Bracco, Annalisa Buffa, Carlotta Giannelli, Rafael Vázquez. Adaptive isogeometric methods with hierarchical splines: An overview. Discrete & Continuous Dynamical Systems - A, 2019, 39 (1) : 241-261. doi: 10.3934/dcds.2019010

[12]

Miguel A. Dumett, Roberto Cominetti. On the stability of an adaptive learning dynamics in traffic games. Journal of Dynamics & Games, 2018, 5 (4) : 265-282. doi: 10.3934/jdg.2018017

[13]

Shunfu Jin, Wuyi Yue, Shiying Ge. Equilibrium analysis of an opportunistic spectrum access mechanism with imperfect sensing results. Journal of Industrial & Management Optimization, 2017, 13 (3) : 1255-1271. doi: 10.3934/jimo.2016071

[14]

A Voutilainen, Jari P. Kaipio. Model reduction and pollution source identification from remote sensing data. Inverse Problems & Imaging, 2009, 3 (4) : 711-730. doi: 10.3934/ipi.2009.3.711

[15]

Haruki Katayama, Hiroyuki Masuyama, Shoji Kasahara, Yutaka Takahashi. Effect of spectrum sensing overhead on performance for cognitive radio networks with channel bonding. Journal of Industrial & Management Optimization, 2014, 10 (1) : 21-40. doi: 10.3934/jimo.2014.10.21

[16]

Paolo Fergola, Marianna Cerasuolo, Edoardo Beretta. An allelopathic competition model with quorum sensing and delayed toxicant production. Mathematical Biosciences & Engineering, 2006, 3 (1) : 37-50. doi: 10.3934/mbe.2006.3.37

[17]

Seungkook Park. Coherence of sensing matrices coming from algebraic-geometric codes. Advances in Mathematics of Communications, 2016, 10 (2) : 429-436. doi: 10.3934/amc.2016016

[18]

Richard L Buckalew. Cell cycle clustering and quorum sensing in a response / signaling mediated feedback model. Discrete & Continuous Dynamical Systems - B, 2014, 19 (4) : 867-881. doi: 10.3934/dcdsb.2014.19.867

[19]

Jan Haškovec, Dietmar Oelz. A free boundary problem for aggregation by short range sensing and differentiated diffusion. Discrete & Continuous Dynamical Systems - B, 2015, 20 (5) : 1461-1480. doi: 10.3934/dcdsb.2015.20.1461

[20]

Min-Fan He, Li-Ning Xing, Wen Li, Shang Xiang, Xu Tan. Double layer programming model to the scheduling of remote sensing data processing tasks. Discrete & Continuous Dynamical Systems - S, 2019, 12 (4&5) : 1515-1526. doi: 10.3934/dcdss.2019104

2018 Impact Factor: 1.469

Metrics

  • PDF downloads (19)
  • HTML views (93)
  • Cited by (0)

Other articles
by authors

[Back to Top]