# American Institute of Mathematical Sciences

August  2017, 11(4): 761-781. doi: 10.3934/ipi.2017036

## On a spatial-temporal decomposition of optical flow

 1 Computational Science Center, University of Vienna, Oskar-Morgenstern Platz 1,1090 Vienna, Austria 2 Johann Radon Institute for Computational and Applied Mathematics, (RICAM), Altenbergerstraẞe 69,4040 Linz, Austria

* Corresponding author: Aniello Raffaele Patrone

Received  July 2015 Revised  April 2017 Published  June 2017

Fund Project: The first author is supported by WWTF

In this paper we present a decomposition algorithm for computation of the spatial-temporal optical flow of a dynamic image sequence. We consider several applications, such as the extraction of temporal motion features and motion detection in dynamic sequences under varying illumination conditions, such as they appear for instance in psychological flickering experiments. For the numerical implementation we are solving an integro-differential equation by a fixed point iteration. For comparison purposes we use a standard time dependent optical flow algorithm, which in contrast to our method, constitutes in solving a spatial-temporal differential equation.

Citation: Aniello Raffaele Patrone, Otmar Scherzer. On a spatial-temporal decomposition of optical flow. Inverse Problems & Imaging, 2017, 11 (4) : 761-781. doi: 10.3934/ipi.2017036
##### References:

show all references

##### References:
$f(x,t)=x(1-x)(1-t)$ from (7). Level lines of $f$ are parametrized by $(\Psi(x,t),t)$
$g(t)=\exp \left\{-\frac{1}{\beta}(1-t)^\beta\right\}$
Color Wheel
${\vec u^{\left( 2 \right)}}$ at different frequencies of rotations: $2$, $4$ and $8 \times$ faster than the original motion frequency. $\alpha^{(1)}=1$, $\alpha^{(2)}=\frac{1}{4}$. The intensity of ${\vec u^{\left( 2 \right)}}$ increases when the frequency of rotation is increased
${\vec u^{\left( 1 \right)}}$: Movement of a Ferris wheel and people walking in the foreground (top left). ${\vec u^{\left( 2 \right)}}$ consists of blinking lights and the reflections of the wheel (top right). The third image (bottom) is a reference frame
The dynamic sequence consists of the smooth (translation like) motion of a cube and an oscillating background. The oscillation has a periodicity of four frames and takes place along the diagonal direction from the bottom left to the top right, moving at a rate of 5% of the frame size in each frame. The proposed model decomposes the motion, obtaining the global movement of the cube in ${\vec u^{\left( 1 \right)}}$ (left) and the background movement exclusively in ${\vec u^{\left( 2 \right)}}$ (right).
The two frames of the flickering sequence containing information (top), the difference between these two frames (down left), and the ${\vec u^{\left( 2 \right)}}$ flow field resulting from the proposed approach (down right). As predicted in Section 3 and Appendix A the ${\vec u^{\left( 1 \right)}}$ component is negligible, instead ${\vec u^{\left( 2 \right)}}$ detects the change of intensity across the blank sheet.
Result with Horn-Schunck
Continuous notation
 $\vec x = (x_1,x_2)$ vector in two-dimensional Euclidean space $\partial_k = \frac{\partial}{\partial x_k}$ differentiation with respect to spatial variable $x_k$ $\partial_t = \frac{\partial}{\partial t}$ differentiation with respect to time $\nabla = (\partial_1, \partial_2)^T$ gradient in space $\nabla_3 = (\partial_1, \partial_2, \partial_t)^T$ gradient in space and time $\nabla \cdot = \partial_1 + \partial_2$ divergence in space $\nabla_3 \cdot = \partial_1 + \partial_2 + \partial_t$ divergence in space and time $\vec{n}$ outward pointing normal vector to $\Omega$ $f$ input sequence $f(\cdot,t)$ movie frame ${\vec u^{\left( i \right)}}$ optical flow module, $i=1,2$ $\vec u = {\vec u^{\left( 1 \right)}} + {\vec u^{\left( 2 \right)}}$ optical flow $u_j^{\left( i \right)}$ $j$-th optical flow component of the $i$-th module $\widehat{u}(\cdot,t) = \int_0^t u(\cdot,\tau)\,{\rm{d}} \tau$ primitive of $u$ $\widehat{\widehat{u}}(\cdot,t) = -\int_t^1 \widehat{u}(\cdot,\tau)\,{\rm{d}} \tau$ 2nd primitive of $u$ -note that $\partial_t \widehat{\widehat{u}}(\cdot,t)=\widehat{u}(\cdot,t)$
 $\vec x = (x_1,x_2)$ vector in two-dimensional Euclidean space $\partial_k = \frac{\partial}{\partial x_k}$ differentiation with respect to spatial variable $x_k$ $\partial_t = \frac{\partial}{\partial t}$ differentiation with respect to time $\nabla = (\partial_1, \partial_2)^T$ gradient in space $\nabla_3 = (\partial_1, \partial_2, \partial_t)^T$ gradient in space and time $\nabla \cdot = \partial_1 + \partial_2$ divergence in space $\nabla_3 \cdot = \partial_1 + \partial_2 + \partial_t$ divergence in space and time $\vec{n}$ outward pointing normal vector to $\Omega$ $f$ input sequence $f(\cdot,t)$ movie frame ${\vec u^{\left( i \right)}}$ optical flow module, $i=1,2$ $\vec u = {\vec u^{\left( 1 \right)}} + {\vec u^{\left( 2 \right)}}$ optical flow $u_j^{\left( i \right)}$ $j$-th optical flow component of the $i$-th module $\widehat{u}(\cdot,t) = \int_0^t u(\cdot,\tau)\,{\rm{d}} \tau$ primitive of $u$ $\widehat{\widehat{u}}(\cdot,t) = -\int_t^1 \widehat{u}(\cdot,\tau)\,{\rm{d}} \tau$ 2nd primitive of $u$ -note that $\partial_t \widehat{\widehat{u}}(\cdot,t)=\widehat{u}(\cdot,t)$
Discrete Notation
 $f= f(r,s,t) \in \mathbb{R}^{M \times N \times T}$ input sequence ${\vec u^{\left( i \right)}} = {\vec u^{\left( i \right)}}(r,s,t;k) \in \mathbb{R}^{M \times N \times T \times K \times 2}$ discrete optical flow approximating the continuous flow ${\vec u^{\left( i \right)}}$ at $(\frac{r}{M-1},\frac{s}{N-1},\frac{t}{T-1})$ $\partial_k^h$ finite difference approximation in direction $x_k$ $\partial_t^h$ finite difference approximation in direction $t$ $\Delta_x=\frac{1}{M-1}$, $\Delta_y=\frac{1}{N-1}$ and $\Delta_t=\frac{1}{T-1}$ Discretization $\hat u_j^{\left( 2 \right)}(r,s,t;k) = \Delta_t \sum_{\tau=1}^t u_j^{\left( 2 \right)}(r,s,\tau;k)$, $j=1,2$ finite difference approximation of $\widehat{u}(\cdot,t)$ $\hat {\hat {u}}_j^{\left( 2 \right)}(r,s,t;k)= - \Delta_t \sum_{\tau = t}^T \hat u_j^{\left( 2 \right)}(r,s,\tau;k)$ finite difference approximation of $\widehat{\widehat{u}}(\cdot,t)$
 $f= f(r,s,t) \in \mathbb{R}^{M \times N \times T}$ input sequence ${\vec u^{\left( i \right)}} = {\vec u^{\left( i \right)}}(r,s,t;k) \in \mathbb{R}^{M \times N \times T \times K \times 2}$ discrete optical flow approximating the continuous flow ${\vec u^{\left( i \right)}}$ at $(\frac{r}{M-1},\frac{s}{N-1},\frac{t}{T-1})$ $\partial_k^h$ finite difference approximation in direction $x_k$ $\partial_t^h$ finite difference approximation in direction $t$ $\Delta_x=\frac{1}{M-1}$, $\Delta_y=\frac{1}{N-1}$ and $\Delta_t=\frac{1}{T-1}$ Discretization $\hat u_j^{\left( 2 \right)}(r,s,t;k) = \Delta_t \sum_{\tau=1}^t u_j^{\left( 2 \right)}(r,s,\tau;k)$, $j=1,2$ finite difference approximation of $\widehat{u}(\cdot,t)$ $\hat {\hat {u}}_j^{\left( 2 \right)}(r,s,t;k)= - \Delta_t \sum_{\tau = t}^T \hat u_j^{\left( 2 \right)}(r,s,\tau;k)$ finite difference approximation of $\widehat{\widehat{u}}(\cdot,t)$
Comparison of squared residuals over space and time $\mathcal{E}$ between Weickert-Schnörr and the proposed method
 Weickert-Schnörr Proposed model Hamburg Taxi 1374.9 1021 RubberWhale 4459.7 3046.8 Hydrangea 8533.3 7647.2 DogDance 9995.4 8217.6 Walking 8077.5 5944.3
 Weickert-Schnörr Proposed model Hamburg Taxi 1374.9 1021 RubberWhale 4459.7 3046.8 Hydrangea 8533.3 7647.2 DogDance 9995.4 8217.6 Walking 8077.5 5944.3
 [1] Raimund Bürger, Gerardo Chowell, Pep Mulet, Luis M. Villada. Modelling the spatial-temporal progression of the 2009 A/H1N1 influenza pandemic in Chile. Mathematical Biosciences & Engineering, 2016, 13 (1) : 43-65. doi: 10.3934/mbe.2016.13.43 [2] Daniil Kazantsev, William M. Thompson, William R. B. Lionheart, Geert Van Eyndhoven, Anders P. Kaestner, Katherine J. Dobson, Philip J. Withers, Peter D. Lee. 4D-CT reconstruction with unified spatial-temporal patch-based regularization. Inverse Problems & Imaging, 2015, 9 (2) : 447-467. doi: 10.3934/ipi.2015.9.447 [3] Yoon-Sik Cho, Aram Galstyan, P. Jeffrey Brantingham, George Tita. Latent self-exciting point process model for spatial-temporal networks. Discrete & Continuous Dynamical Systems - B, 2014, 19 (5) : 1335-1354. doi: 10.3934/dcdsb.2014.19.1335 [4] Thomas Schuster, Joachim Weickert. On the application of projection methods for computing optical flow fields. Inverse Problems & Imaging, 2007, 1 (4) : 673-690. doi: 10.3934/ipi.2007.1.673 [5] Umberto Mosco. Impulsive motion on synchronized spatial temporal grids. Discrete & Continuous Dynamical Systems - A, 2017, 37 (12) : 6069-6098. doi: 10.3934/dcds.2017261 [6] Lukas F. Lang, Otmar Scherzer. Optical flow on evolving sphere-like surfaces. Inverse Problems & Imaging, 2017, 11 (2) : 305-338. doi: 10.3934/ipi.2017015 [7] Andrey Shishkov. Large solutions of parabolic logistic equation with spatial and temporal degeneracies. Discrete & Continuous Dynamical Systems - S, 2017, 10 (4) : 895-907. doi: 10.3934/dcdss.2017045 [8] Daijun Jiang, Hui Feng, Jun Zou. Overlapping domain decomposition methods for linear inverse problems. Inverse Problems & Imaging, 2015, 9 (1) : 163-188. doi: 10.3934/ipi.2015.9.163 [9] Iordanka N. Panayotova, Pai Song, John P. McHugh. Spatial stability of horizontally sheared flow. Conference Publications, 2013, 2013 (special) : 611-618. doi: 10.3934/proc.2013.2013.611 [10] Tom Goldstein, Xavier Bresson, Stan Osher. Global minimization of Markov random fields with applications to optical flow. Inverse Problems & Imaging, 2012, 6 (4) : 623-644. doi: 10.3934/ipi.2012.6.623 [11] Xavier Litrico, Vincent Fromion. Modal decomposition of linearized open channel flow. Networks & Heterogeneous Media, 2009, 4 (2) : 325-357. doi: 10.3934/nhm.2009.4.325 [12] Lijian Jiang, Yalchin Efendiev, Victor Ginting. Multiscale methods for parabolic equations with continuum spatial scales. Discrete & Continuous Dynamical Systems - B, 2007, 8 (4) : 833-859. doi: 10.3934/dcdsb.2007.8.833 [13] Liselott Flodén, Jens Persson. Homogenization of nonlinear dissipative hyperbolic problems exhibiting arbitrarily many spatial and temporal scales. Networks & Heterogeneous Media, 2016, 11 (4) : 627-653. doi: 10.3934/nhm.2016012 [14] Jiao-Yan Li, Xiao Hu, Zhong Wan. An integrated bi-objective optimization model and improved genetic algorithm for vehicle routing problems with temporal and spatial constraints. Journal of Industrial & Management Optimization, 2017, 13 (5) : 1-18. doi: 10.3934/jimo.2018200 [15] Julie Lee, J. C. Song. Spatial decay bounds in a linearized magnetohydrodynamic channel flow. Communications on Pure & Applied Analysis, 2013, 12 (3) : 1349-1361. doi: 10.3934/cpaa.2013.12.1349 [16] Lori Badea. Multigrid methods for some quasi-variational inequalities. Discrete & Continuous Dynamical Systems - S, 2013, 6 (6) : 1457-1471. doi: 10.3934/dcdss.2013.6.1457 [17] Zalman Balanov, Carlos García-Azpeitia, Wieslaw Krawcewicz. On variational and topological methods in nonlinear difference equations. Communications on Pure & Applied Analysis, 2018, 17 (6) : 2813-2844. doi: 10.3934/cpaa.2018133 [18] Edson Pindza, Francis Youbi, Eben Maré, Matt Davison. Barycentric spectral domain decomposition methods for valuing a class of infinite activity Lévy models. Discrete & Continuous Dynamical Systems - S, 2019, 12 (3) : 625-643. doi: 10.3934/dcdss.2019040 [19] O. Chadli, Z. Chbani, H. Riahi. Recession methods for equilibrium problems and applications to variational and hemivariational inequalities. Discrete & Continuous Dynamical Systems - A, 1999, 5 (1) : 185-196. doi: 10.3934/dcds.1999.5.185 [20] Raffaella Servadei, Enrico Valdinoci. Variational methods for non-local operators of elliptic type. Discrete & Continuous Dynamical Systems - A, 2013, 33 (5) : 2105-2137. doi: 10.3934/dcds.2013.33.2105

2018 Impact Factor: 1.469