Visual Tracking Using Sparse Coding and Earth Mover's Distance

An efficient iterative Earth Mover's Distance (iEMD) algorithm for visual tracking is proposed in this paper. The Earth Mover's Distance (EMD) is used as the similarity measure to search for the optimal template candidates in feature-spatial space in a video sequence. The local sparse representation is used as the appearance model for the iEMD tracker. The maximum-alignment-pooling method is used for constructing a sparse coding histogram which reduces the computational complexity of the EMD optimization. The template update algorithm based on the EMD is also presented. When the camera is mounted on a moving robot, e.g., a flying quadcopter, the camera could experience a sudden and rapid motion leading to large inter-frame movements. To ensure that the tracking algorithm converges, a gyro-aided extension of the iEMD tracker is presented, where synchronized gyroscope information is utilized to compensate for the rotation of the camera. The iEMD algorithm's performance is evaluated using eight publicly available videos from the CVPR 2013 dataset. The performance of the iEMD algorithm is compared with eight state-of-the-art tracking algorithms based on relative percentage overlap. Experimental results show that the iEMD algorithm performs robustly in the presence of illumination variation and deformation. The robustness of this algorithm for large inter-frame displacements is also illustrated.

1. Introduction

Visual tracking is an important problem for new robotics applications. The information generated from the sequence of images by the tracking algorithm can be utilized by vehicle navigation, human-robot interaction, and motion-based recognition algorithms (Dani et al., 2013; Ravichandar and Dani, 2015; Chwa et al., 2016). Visual tracking algorithms provide important information for visual simultaneous localization and mapping (SLAM), structure from motion (SfM), and vision-based control (Marchand and Chaumette, 2005; Marchand et al., 2005; Comport et al., 2006; Davison et al., 2007; Dani et al., 2012; Yang et al., 2015).

Image-based tracking algorithms are categorized as point tracking, kernel tracking, or silhouette tracking (Yilmaz et al., 2006). Distinguishing features, such as color, shape, and region are selected to identify objects for visual tracking. Modeling the object adapts to the slowly changing appearance is challenging, due to the illumination variants, object deformation, occlusion, motion blur, or background clutters. Supervised or unsupervised online learning algorithms are often used to robustly find and update the distinguishing features of the object, such as using variance ratios of the feature value's log likelihood (Collins et al., 2005), the online Ada-boost feature selection method (Grabner and Bischof, 2006), and incremental learning (Ross et al., 2008).

Approaches in visual tracking could be generally classified into two groups, either generative methods or discriminative methods. For generative methods, the tracked object is modeled based on the selected features, such as the color histogram, sparse coding representation, or kernels. Then, correspondence or similarity measurement between the target and the candidate across frames is constructed. Similarity measurements are derived through several methods, such as the Normalized Cross Correlation (NCC) (Bolme et al., 2010; Zhu et al., 2016), the Earth Mover's Distance (EMD) (Zhao et al., 2010; Karavasilis et al., 2011; Oron et al., 2012; Tahri et al., 2016), the Bhattacharyya Coefficient (BC) (Comaniciu et al., 2003) and point-to-set distance metric (Wang et al., 2015, 2016). Location of the candidate object in the consecutive frames is estimated by using the Kalman filter, particle filter or gradient descent method. Discriminative methods regard tracking as a classification problem and build a classifier or ensemble of classifiers to distinguish the object from the background. Representative classification tracking algorithms are the structured Support Vector Machine (SVM) (Hare et al., 2011), Convolutional Neural Nets (Li et al., 2016). Ensemble based algorithms such as ensemble tracking (Avidan, 2007), multiple instance learning (MIL) (Babenko et al., 2011), and online boosting tracker (Grabner and Bischof, 2006).

In order to robustly track moving objects in challenging situations, many tracking frameworks are proposed. Tracking algorithms with Bayesian filtering are developed to track moving objects. These algorithms can handle complete occlusion (Zivkovic et al., 2009). The non-adaptive methods, usually only model the object from the first frame. Although less error prone to occlusions and drift, they are hard to track the object undergoing appearance variations. However, adaptive meth ods are usually prone to drift because they rely on self updates of an online learning method. In order to deal with this problem, combining adaptive methods with the complementary tracking approaches leads to more stable results. For example, parallel robust online simple tracking (PROST) framework combines three different trackers (Santner et al., 2010): tracking-learning-detection (TLD) framework uses P-N experts to make the decision on the location of the moving object, based on the results from the Median-Flow tracker and detectors (Kalal et al., 2012), and online adaptive hidden Markov model for multi-tracker fusion (Vojir et al., 2016).

An EMD-based tracker using color histogram [iEMD(CH)] as an appearance model and its fusion with gyroscope information is presented in our prior related work (Yao et al., 2016; Yao and Dani, 2017). However, color histogram model is not robust to appearance changes. Also, the template update algorithm is not used in our prior related work. The sparse coding appearance model is based on a dictionary of templates consisting of the appearance variations of the target. The sparse coding appearance model has been used in literature and has shown robust performance in various tracking algorithms (Zhang et al., 2013). In this paper, we develop a generative tracking method using sparse coding appearance model along with EMD as a similarity measure. An adaptive template update algorithm is also developed to update the apprearance model during tracking to handle the appearance variations. Gyroscope information is used to aid the initialization of the EMD optimization. Specifically, the contributions of the paper are

• The maximum-alignment-pooling method for local sparse coding is used to build a histogram of appearance model. A template update algorithm is used to adaptively change the appearance model by an exponential rule based on EMD measure. An iEMD tracking algorithm is developed based on this local sparse coding representation of the appearance model. It is shown using videos from publicly available benchmark datasets that the iEMD tracker shows good performance in terms of percentage overlap compared to the state-of-the-art trackers available in literature.

• Gyro-measurements are used to compensate for the pan, tilt, and roll of the camera. Then the iEMD visual tracking algorithm is used to track the target after compensating for the movement of the camera. By this method, the convergence of the algorithm is ensured, thus providing a more robust tracker which is more capable of real-world tracking tasks.

The paper is organized as follows. Related work on the computation of the EMD and its application for visual tracking is illustrated in section 2. In section 3, the iEMD algorithm for visual tracking is developed. In section 4, the target is modeled as the sparse coding histogram. For the sparse coding histogram, the maximum-alignment-pooling method is proposed to represent the local image patches. In section 5, two extensions of the iEMD algorithm that includes the template update method, and the method of using the gyroscope data for ego-motion compensation are discussed. In section 6, the iEMD tracker is validated on eight publicly available datasets, and the comparisons with eight state-of-the-art trackers are shown. Experimental results using the gyro-aided iEMD algorithm are compared with tracking results without gyroscope information. The conclusions are given in section 7.

2. Related Work

Sparse coding has been successfully applied to model the target in visual tracking (Zhang et al., 2013). In sparse coding for visual tracking, the largest sum of the sparse coefficients or the smallest reconstruction error is used as the metric to find the target from the candidate templates using particle filter (Mei and Ling, 2009; Jia et al., 2016). The sparse coding process is usually the L₁ norm minimization problem, which makes the sparse representation and dictionary learning computationally expensive. To reduce the computational complexity, the sparse representation as the appearance model is combined with the Mean-shift (Liu et al., 2011) or Mean-transform method (Zhang and Hong Wong, 2014). After a small number of iterations by these methods, the maximum value of the Bhattacharyya coefficient corresponding to the best candidate is obtained.

In real-world tracking applications, variations in appearance are a common phenomenon caused by illumination changes, moderate pose changes or partial occlusions. The Earth Mover's Distance (EMD) as a similarity measure, also known as 1-Wasserstein distance (Guerriero et al., 2010; Baum et al., 2015), is robust to these situations (Rubner et al., 2000). However, the major problem with the EMD is its computational complexity. Several algorithms for the efficient computation of the EMD are proposed. For example, the EMD-L₁ algorithm is used for histogram comparison (Ling and Okada, 2007) and the EMDs are computed with the thresholded ground distances (Pele and Werman, 2009). In the context of visual tracking, although the EMD has the merit of being robust to moderate appearance variations, the efficiency of the computation is still a problem. Since solving the EMD is a transportation problem—a linear programming problem (Rubner et al., 2000), the direct differential method cannot be used. There are some efforts to employ the EMD for object tracking. The Differential Earth Mover's Distance (DEMD) algorithm (Zhao et al., 2010) is first proposed for visual tracking, which adopts the sensitivity analysis to approximate the derivative of the EMD. However, the selection of the basic variables and the process of identifying and deleting the redundant constraints still affect the efficiency of the algorithm (Zhao et al., 2010). The DEMD algorithm combined with the Gaussian Mixture Model (GMM), which has fewer parameters for EMD optimization, is proposed in Karavasilis et al. (2011). The EMD as the similarity measure combined with the particle filter for visual tracking is proposed in Oron et al. (2012). In this paper, the sparse coding is used along with EMD similarity measure for the visual tracking. To the best of our knowledge, this is the first work that combines sparse coding representation with the EMD similarity measure for visual tracking.

The success of the gradient descent based tracking algorithm depends on the assumption that the object motion is smooth and contains only small displacements (Yilmaz et al., 2006). However, in practice, this assumption is always violated due to the abrupt rotation and shaking movement of the camera mounted on a mobile robot, such as a flying quadcopter. Efforts have been made to combine the gyroscope data with tracking algorithms, such as the Kanade-Lucas-Tomasi (KLT) tracker or the MI tracker (Hwangbo et al., 2011; Park et al., 2013; Ravichandar and Dani, 2014). In our paper, to robustly track a static object using a moving camera, gyroscope data are directly utilized to estimate the initial location of the static object. When both the camera and the object being tracked are in motion, the gyroscope sensor data are utilized to compensate for the rotation of the camera, because rotation has a greater impact on the positional changes compared with the translation in video frames. Then, the iEMD tracking algorithm is applied to track the moving object. The robustness of the tracking algorithm is improved due to the compensation of the camera's ego-motion. Therefore, our method makes the EMD tracker more robust to this situation.

3. Iterative EMD Tracking Algorithm

In the context of visual tracking, first a feature space is chosen to characterize the object, then, the target model and the candidate model are built in the feature-spatial space. The probability density functions (histograms) representing the target model and the candidate model are (Comaniciu et al., 2003)

$\text{target model}:\widehat{p}={\left\{{\widehat{p}}_u\right\}}_{u=1,\dots ,N_T}\text{ and }{\displaystyle {\sum }_1^{N_T}{\widehat{p}}_u}=1$

$\text{candidate model}:\widehat{q}(y)={\left\{{\widehat{q}}_v(y)\right\}}_{u=1,\dots ,N_C}\text{and }{\displaystyle {\sum }_1^{N_C}{\widehat{q}}_u}(y)=1,$

where ${\widehat{p}}_u$ is the weight of the uth bin of the target model $\widehat{\text{p}}$, assuming the center of the template target is at (0, 0), ${\widehat{q}}_v$ is the weight of the vth bin of the candidate model $\widehat{\text{q}}(\text{y})$, assuming the center of the template candidate is at y, N_T and N_C are the numbers of the bins.

Based on the target model and the candidate model, the dissimilarity function is denoted as $f(\widehat{\text{p}},\widehat{\text{q}}(\text{y}))$. The optimization problem for tracking is to estimate the optimal displacement $\widehat{\text{y}}$ which gives the smallest value of $f(\widehat{\text{p}},\widehat{\text{q}}(\text{y}))$. Thus, the optimization problem is formulated as

$\begin{array}{ll}\widehat{\text{y}}=arg{\displaystyle \underset{\text{y}}{min}}f(\widehat{\text{p}},\widehat{\text{q}}(\text{y}))& (1)\end{array}$

In (1), the center of the template target is assumed to be positioned at (0, 0), and the center of the template candidate is at y. The goal is to find the candidate model located at $\widehat{\text{y}}$ that gives the smallest value of the dissimilarity function $f(\widehat{\text{p}},\widehat{\text{q}}(\text{y}))$. The differential tracking approaches are usually applied to solve this optimization problem, with the assumption that the displacement of the target between two consecutive frames is very small.

The optimization problem in (1) is solved using the iEMD algorithm as described in the following sub-sections. The iEMD algorithm iterates between finding the smallest EMD between template target and the template candidate based on the current position y_k by the transportation-simplex method (see section 3.2 for details) and finding the best position y_k+1 leading to the smallest EMD by gradient method (see section 3.3 for details).

3.1. EMD as a Similarity Measure

In this section, the EMD between the target model $\widehat{\text{p}}$ and the candidate model $\widehat{\text{q}}(\text{y})$ is used as the similarity measure. Solving the EMD is a transportation problem—a linear programming problem as shown in Figure 1. Intuitively, given the target model and the candidate model, each bin of both models are cross compared. The costs between the bins from two different models are predefined. Then the EMD is considered as the smallest overall cost of sending the weights of one bin from the target model to another bin of the candidate model. The EMD is defined as (Rubner et al., 2000)

$\begin{array}{ll}{D}^{\star }(f_{uv}(\widehat{\text{p}},\widehat{\text{q}}(\text{y})))\triangleq {\displaystyle \underset{f_{uv}}{min}}\left({\displaystyle \sum _{u=1}^{N_T}}{\displaystyle \sum _{v=1}^{N_C}}{d}_{uv}{f}_{uv}(\widehat{\text{p}},\widehat{\text{q}}(\text{y}))\right)& (2)\end{array}$

subject to

$\begin{array}{ll}{\displaystyle \sum _{u=1}^{N_T}}{f}_{uv}(\widehat{\text{p}},\widehat{\text{q}}(\text{y}))=w_{C,v},1\le v\le N_C& (3)\end{array}$

$\begin{array}{ll}{\displaystyle \sum _{v=1}^{N_C}}{f}_{uv}(\widehat{\text{p}},\widehat{\text{q}}(\text{y}))=w_{T,u},1\le u\le N_T& (4)\end{array}$

$\begin{array}{ll}{\displaystyle \sum _{u=1}^{N_T}}{\displaystyle \sum _{v=1}^{N_C}}{f}_{uv}(\widehat{\text{p}},\widehat{\text{q}}(\text{y}))=1& (5)\end{array}$

$\begin{array}{ll}{f}_{uv}(\widehat{\text{p}},\widehat{\text{q}}(\text{y}))\ge 0,1\le u\le N_T,1\le v\le N_C& (6)\end{array}$

where D^⋆ is the optimal solution to this transportation problem, $f_{uv}(\widehat{\text{p}},\widehat{\text{q}}(\text{y}))$ is the flow (weight) from the uth bin of $\widehat{\text{p}}$ to the vth bin of $\widehat{\text{q}}(\text{y})$, d_uv is the ground distance (cost) between the uth and the vth bins, the subscript T denotes the object target and C is the object candidate, w_{T, u} is the weight from the uth bin of $\widehat{\text{p}}$, and w_{C, v} is the weight from the vth bin of $\widehat{\text{q}}(\text{y})$.

FIGURE 1

Figure 1. EMD comparison of the two templates (No permission is required from the copyright holders and/or the depicted individuals for the use of these images. The original images are obtained from the EC Funded CAVIAR project/IST 2001 37540, found at http://homepages.inf.ed.ac.uk/rbf/CAVIAR/).

3.2. EMD as a Function of Weights

Writing the above equation set (2–6) in a matrix form as

$\begin{array}{l}{D}^{\star }=\underset{f}{\text{min}}{d}^{T}f\\ \text{s}.\text{t}.Hf=w;f>0& (7)\end{array}$

where the $\text{d}={\left[d_{11},\cdots d_{1N_C},\cdots d_{N_{T}1},\cdots d_{N_T{N}_C}\right]}^T\in {ℝ}^{N_T{N}_C}$ is the ground distance vector, $f={[f_{11},\cdots f_{1,N_C},\cdots f_{N_{T}1,}\cdots f_{N_T{N}_C}]}^T\in {ℝ}^{N_T{N}_C}$is the flow vector, $w={[w_{N_C}^T,w_{N_T}^T]}^T\in {ℝ}^{N_T+N_C}$ is the weight vector consisting of the weight vectors $w_{N_C}\in {ℝ}^{N_C}$ from $\widehat{\text{q}}(\text{y})$ and $w_{N_T}\in {ℝ}^{N_T}$ from $\widehat{\text{p}}$, and $H\in {ℝ}^{(N_T+N_C)\times N_T{N}_C}$ is the matrix which consists of 0 and 1s.

In order to relate the EMD with the weight vector, the above primal problem in (7) is restated in its dual form as (Dantzig and Thapa, 2006)

$\begin{array}{l}{D}^{\star }=\underset{\pi }{\text{max }}{w}^T\pi \\ \text{s}.\text{t}.H^T\pi \le d& (8)\end{array}$

where $\pi \in {ℝ}^{N_T+N_C}$ is a vector of variables to be optimized in the dual problem. By solving this dual problem in (8), the optimal solution D^⋆ is calculated and directly represented as the linear equation of weights. However, considering the computation efficiency, the optimal solution (EMD) is first calculated from the primal problem in (7) using the transportation-simplex method, and then the EMD is represented as the function of the weights by the matrix transformation.

Using the transportation-simplex method (Rubner et al., 2000), the optimal solution to the EMD problem in (7) is calculated. The transportation-simplex method is a streamlined simplex algorithm, which is built on the special structure of the transportation problem. In order to reduce the number of iterations of the transportation-simplex method, the Russell's method is used to compute the initial basic feasible solution (Rubner et al., 2000; Ling and Okada, 2007).

The computation of the EMD is a transportation problem, which has exactly N_T+N_C−1 basic variables $f_B\in {ℝ}^{N_T+N_C-1}$, and each constraint is a linear combination of the other N_T+N_C−1 constraints, which could be considered as redundant and discarded (Dantzig and Thapa, 2006). Based on the optimal solution to the linear programming problem, the flow vector is separated into basic variables and non-basic variables $f={[f_B^T,f_{NB}^T]}^T\in {ℝ}^{N_T{N}_C},$, and the ground distance vector d and H will be transformed as $d={[d_B^T,d_{NB}^T]}^T\in {ℝ}^{N_T{N}_C}$ and $H={[H_B,H_{NB}]}^T\in {ℝ}^{(N_T+N_C)\times N_T{N}_C},$, where $d_B\in {ℝ}^{N_T+N_C-1}$, and $H_B\in {ℝ}^{(N_T+N_C)\times (N_T+N_C-1)}.$. In order to derive the EMD as a function of the weights of the candidate model, the matrix transformation is performed. First, the last row of the constraint matrices (7) is deleted which is considered as the redundant constraint and then the matrices H_B, H, and w are formulated as $H_B^{\ast }\in {ℝ}^{(N_T+N_C-1)\times (N_T+N_C-1)}$, $H^{\ast }={[H_B^{\ast },H_{NB}^{\ast }]}^T\in {ℝ}^{(N_T+N_C-1)\times N_T{N}_C}$ and $w^{\ast }={[w_{N_C}^T,w_{N_T-1}^T]}^T\in {ℝ}^{N_T+N_C-1}.$.

The problem in (7) is reformulated based on the optimal solution as

$\begin{array}{ll}{D}^{\star }-{{\displaystyle d_B}}^T{f}_B-{{\displaystyle d_{NB}}}^T{f}_{NB} =0\hfill & (9)\end{array}$

$\begin{array}{ll}{H}_B^{\ast }{f}_B+H_{NB}^{\ast }{f}_{NB}=w^{\ast }& (10)\end{array}$

Left multiplying (10) with $H_B^{\ast -1}$ yields

$\begin{array}{ll}{f}_B+H_B^{\ast -1}{H}_{NB}^{\ast }{f}_{NB}=H_B^{\ast -1}{w}^{\ast }& (11)\end{array}$

Left multiplying (11) by $d_B{}^T$ and adding the resultant to (9) gives

$\begin{array}{ll}{D}^{\star }+(-{{\displaystyle d_{NB}}}^T+M{H}_{NB}^{\ast })f_{NB}=M{w}^{\ast }& (12)\end{array}$

where $M=d_B{}^T{H}_B^{\ast -1}$ is a N_C+N_T−1-dimensional vector. Since f_NB = 0_{NTNC−NT−NC+1}, using (12) the EMD D^⋆ is given by

$\begin{array}{ll}{D}^{\star }=M{[w_{N_C}^T,0]}_{N_C+N_T-1}^T+M{[0,w_{N_T-1}^T]}_{N_C+N_T-1}^T& (13)\end{array}$

3.3. Gradient Method to Find the Template Displacement

Based on the Equation (13), the gradient method is utilized to find the displacement y of the target candidate as

$\begin{array}{ll}\frac{\partial D^{\star }}{\partial y}=M{\left[\frac{\partial w_{N_C}^T}{\partial y},0\right]}_{N_C+N_T-1}^T& (14)\end{array}$

The optimal location $\widehat{\text{y}}$ of the template candidate $\widehat{\text{q}}(\text{y})$ is found by iteratively performing: (1) calculate the smallest EMD and reformulate it as (13); (2) search for the new location of the template candidate along the direction of (14). When the EMD no longer decreases, the iteration stops. By this method, the best match of the template target and the template candidate will be found. The EMD plays three roles in this algorithm: (1) it provides a metric of the matching between the template target and the template candidate; (2) it assigns more weights to the best matches between the histogram bins and assigns smaller weights or no weights to unmatched bins by linear optimization; (3) matched bins are used for finding the location of the template candidate, and the gradient vector of the EMD for searching the optimal displacement is calculated.

The pseudo-code for the iEMD tracking algorithm is given in Algorithm 1.

ALGORITHM 1

Algorithm 1: iEMD tracking algorithm.

4. Target Modeling Based on Histograms of Sparse Codes

Sparse codes histogram (SCH) has been widely used as feature descriptors in many fields (Zhang et al., 2013). Given the image set of the first L image templates from a video, a set of K overlapped local image patches are sampled by a sliding window of size m×n from each template to build a dictionary Φ∈ℝ^{(mn) × (LK)}. Each column of Φ is a basis vector, which is a vectorized local image patch extracted from the set of image templates. The basis vectors are overcomplete where mn < LK. Similarly, for a given image template target I, a set of overlapped local image patches $\text{E}=\left\{{ϵ}_r|{ϵ}_r\in {ℝ}^{(mn)\times 1},r=1\cdots J\right\}$ are sampled by the same sliding window of size m×n with the step size of one pixel. Each image patch ϵ_r, which represents one fixed part of the target object, can be encoded as a linear combination of a few basis vectors of the dictionary Φ as follows

$\begin{array}{ll}{ϵ}_r=\Phi {\text{a}}_r+\text{n}& (15)\end{array}$

where ${\text{a}}_r\in {ℝ}^{(LK)\times 1}$ is the coefficient vector which is sparse and n∈ℝ^{(mn) × 1} is the noise vector. The coefficient a_r is computed by solving the following L₁ norm minimization problem (Zhang et al., 2013; Mairal et al., 2014)

$\begin{array}{l}\underset{a_r}{\text{min}}{\Vert {ϵ}_r-\Phi a_r\Vert }_2^2+\lambda {\Vert a_r\Vert }_1\\ \text{s}.\text{t}.{(a_r)}_k\ge 0,\forall k& (16)\end{array}$

where ${\text{a}}_r={\left[\begin{array}{ccccccc}{a}_{11},& \cdots ,& a_{1K},& \cdots ,& a_{L1},& \cdots ,& a_{LK}\end{array}\right]}^T$ is the sparse coefficients of the local patch, a_ij corresponds to the jth patch of the ith image template of the dictionary, and λ is the Lagrange multiplier.

Once a solution to (16) is obtained, the maximum-alignment-pooling method is used to construct the sparse coding histograms. Combining the coefficients corresponding to the dictionary patches that have the same locations in the template using ${ā}_j=\sum _{i=1}^L{a}_{ij}$ (Jia et al., 2012), a new vector ${\bar{\text{a}}}_r={\left[\begin{array}{ccc}{ā}_1,& \cdots ,& {ā}_j\end{array}\right]}^T\in {ℝ}^{K\times 1}$ is formulated. The weight of the rth local image patch ϵ_r in the histogram of sparse codes is computed by using ${\widehat{p}}_{ru}={{\bar{\text{a}}}_r}_{\infty }$. The ${\widehat{p}}_{ru}$ value corresponds to the uth image patch from ${\bar{\text{a}}}_r$. With J local image patches from the template target, the histogram is constructed as

$\begin{array}{ll}\widehat{\text{p}}={\left[\begin{array}{c}{\widehat{p}}_{11},\cdots ,{\widehat{p}}_{ru},\cdots ,{\widehat{p}}_{JK}\end{array}\right]}^T\in {ℝ}^{J\times 1}& (17)\end{array}$

In the spatial space, the Epanechnikov kernel is used to represent the template. The Epanechnikov kernel (Comaniciu et al., 2003) is an isotropic kernel with a convex profile which assigns smaller weights to pixels away from the center. Given the target histogram $\widehat{\text{p}}$ in (17), the isotropic kernel is applied to generate the histograms of target weighted by the spatial locations. The weights of the histogram of the target w_{T, u} are computed using

$\begin{array}{ll}{w}_{T,u}=\gamma {\displaystyle \sum _{r=1}^J}\left(1-{\frac{{\text{c}}_r}{h}}^2\right)\left|{\widehat{p}}_{ru}\right|& (18)\end{array}$

where c_r is the center of the rth image patch of the template target, h is template size and γ is the normalization constant. The candidate histogram $\widehat{\text{q}}$ is built in the same way as $\widehat{\text{p}}$. An isotropic kernel is applied to the elements of the $\widehat{\text{q}}$ for generating the histogram of candidate with spatial locations. The weights of the candidate histogram w_{C, v}(x_i−y) are computed using

$\begin{array}{ll}{w}_{C,v}(x_i-y)=\gamma {\displaystyle \sum _{r=1}^J\left(1-{\Vert \frac{c_r-y}{h}\Vert }^2\right)}\left|{\widehat{q}}_{rv}\right|& (19)\end{array}$

where y is the displacement of the rth image patch of the template candidate. The ground distance d_uv for the EMD in (2) is defined by

$\begin{array}{ll}{d}_{uv}=\alpha {\Vert {ϵ}_u-{ϵ}_v\Vert }_2^2+\left(1-\alpha \right){\Vert c_u-c_v\Vert }_2^2& (20)\end{array}$

where α ∈ (0, 1) is the weighting coefficient, ${ϵ}_u\in {ℝ}^{(mn)\times 1}$, ${ϵ}_v\in {ℝ}^{(mn)\times 1}$ are the vectors of the normalized pixel values of the image patch from the target and candidate templates, sampled in the same way as the image patches from the dictionary, and c_u, c_v are the corresponding centers of the image patches.

5. Extensions of The Tracking Algorithm

5.1. Template Update

In order to make the tracker robust to significant appearance variations during long video sequences, the outdated templates in the dictionary should be replaced with the recent ones. To adapt to the appearance variations of the target and alleviate the drift problem only the latest template in dictionary is replaced based on the weight ω_i, which is computed by

$\begin{array}{ll}{\omega }_i={\gamma }_0^{\Delta i}\times exp(-D_k^{*})& (21)\end{array}$

where ω_i is the weight associated with the template, γ₀ is a constant, Δi is the time elapsed since the dictionary was last updated measured in terms of image index k and $D_k^{*}$ is the EMD value corresponding to the template I_k.

If the weight of the current template based on (21) is smaller than the weight of the latest template in the dictionary, the template is replaced with the current one. In order to avoid the errors and noises affecting the dictionary update algorithm, the reconstructed template is used to replace the one in the dictionary. Firstly, the following problem is solved in order to recompute the sparse code coefficients, a_k,

$\begin{array}{ll}\underset{a_k}{\text{min}}{\Vert I_k-\left[\begin{array}{cc}{\Phi }_{\text{T}}& I_{mn\times mn}\end{array}\right]a_k\Vert }_2^2+\lambda {\Vert a_k\Vert }_1& (22)\end{array}$

where ${\Phi }_{\text{T}}\in {ℝ}^{(mn)\times K}$ is a dictionary formed using the vectorized template image with the size m×n as columns, I_mn×mn is the identity matrix, ${\text{a}}_k\in {ℝ}^{K+mn}$ is the vector of the sparse coding coefficients, and λ is the Lagrange multiplier (cf., Jia et al., 2012). Then the reconstructed template is calculated using ${\Phi }_{\text{T}}{a}_k^{*}$, where ${\text{a}}_k^{*}\in {ℝ}^K$ is computed using components of a_k corresponding to the dictionary. The reconstructed template is used to replace the latest template in the dictionary. The detailed steps of the update scheme are given in Algorithm 2.

ALGORITHM 2

Algorithm 2: Template update procedure.

5.2. Gyroscope Data Fusion for Rotation Compensation

The general idea of the gyro-aided iEMD tracking algorithm is combining the image frames from the camera with the angular rate generated by the gyroscope for visual tracking. Synchronization of the camera and the gyroscope in time is required. The spatial relationship between the camera and the gyroscope must also be pre-calibrated. Then, the angular rate generated by the gyroscope is applied to compensate for the ego-motion of the camera. After the compensation of the ego-motion of the camera, the iEMD tracker is applied for tracking. In this section, details of the gyro-aided iEMD tracking algorithm are explained.

When a camera is mounted on a moving robot, the motion of the camera will cause a large displacement of the target between two consecutive frames. If the displacement is larger than the convergence region, the tracking algorithm will become susceptible to the large appearance changes and fail (Comaniciu et al., 2003; Hwangbo et al., 2011; Ravichandar and Dani, 2014). In order to improve the robustness of the tracking algorithm, the displacement caused by the camera rotation is estimated and compensated by fusing the data from the gyroscope, which is a commonly used sensor on flying robots. The rotation of the camera causes a larger displacement of the target compared with the translation movement in video-rate frames. Thus, the translation is neglected here.

The gyroscope provides the angular rate along three axes, which measure the pan, tilt, and roll of small time intervals Δt. In the case of pure rotation without translation, the angular rate ω_y is obtained along three axes x, y and z. Let q(k), q(k+1)∈ℍ denote the quaternion of two frames k and k+1 during time Δt, the relationship between them is given as (cf. Spong et al., 2006)

$\begin{array}{ll}q(k+1)=q(k)+\frac{1}{2}\Omega (\omega )·q(k)·\Delta t& (23)\end{array}$

where Ω(ω) is the skew-symmetric matrix of ω as

$\begin{array}{ll}\Omega (\omega )=\left[\begin{array}{cc}0& -{\omega }^T\\ \omega & -{\left[\omega \right]}_{\times }\end{array}\right]& (24)\end{array}$

After the quaternion q(k+1) = m+ai+bj+ck is normalized and updated, the rotation matrix $R_k^{k+1}$ is calculated as

$\begin{array}{ll}{R}_k^{k+1}=\left[\begin{array}{ccc}1-2b^2-2c^2& 2ab-2cm& 2ac+2bm\\ 2ab+2cm& 1-2a^2-2c^2& 2bc-2am\\ 2ac-2bm& 2bc+2am& 1-2a^2-2b^2\end{array}\right]& (25)\end{array}$

Thus, the estimated homography matrix between two templates is estimated by

$\begin{array}{ll}{{H}}_{\text{gyro}}=K{R}_k^{k+1}{K}^{-1}& (26)\end{array}$

where, K is the intrinsic camera calibration matrix that is accessed by calibrating the camera. The homography matrix is update d to the newest frame location $p(k+1)={\left[x_c,y_c,1\right]}^T$, where (x_c, y_c) is the center point of the template, based on the following equations:

$\begin{array}{ll}{{H}}^{k+1}={{H}}^k{{H}}_{\text{gyro}}^{-1}& (27)\end{array}$

$\begin{array}{ll}p(k+1)={{H}}^{k+1}·p(k)& (28)\end{array}$

for the first frame, ${{H}}^0={\text{I}}_{3\times 3}$. This new location $p(k+1)={\left[x_c,y_c,1\right]}^T$ is then used as the initial guess of the object candidate.

The pseudo-code for gyro-aided iEMD algorithm is given in Algorithm 3.

ALGORITHM 3

Algorithm 3: Gyro-aided iEMD tracking algorithm.

6. Experiments

In this section, the iEMD algorithm is validated on real datasets. The algorithm is implemented in MATLAB R2015b, the C code in Rubner et al. (2000) is adopted for the EMD calculation, and the software in Mairal et al. (2014) is used for sparse modeling. The platform is Microsoft Windows 7 professional with Intel(R) Core(TM) i5-4590 CPU. Eight publicly available datasets are chosen to validate the iEMD tracking algorithm. The main attributes of the video sequences are summarized in Table 1. The Car2, Walking, Woman, Subway, Bolt2, Car4, Human8, and Walking2 sequences are from the visual tracker benchmark (Wu et al., 2013) (CVPR 2013, http://www.visual-tracking.net). The length of the sequences varies between 128 and 913 frames with one object being tracked in each frame.

TABLE 1

Table 1. The main attributes of the video sequences.

The tracker is initialized with the ground-truth bounding box of the target in the first frame. Then the tracking algorithm runs till the end of the sequence and generates a series of the tracked bounding boxes. Tracking results from consecutive frames are compared with the ground truth bounding boxes provided by this dataset. The relative overlap measure is used to evaluate this algorithm as (Wu et al., 2013)

$\begin{array}{ll}\text{overlap}=\frac{R_{tr}\cap R_{gt}}{R_{tr}\cup R_{gt}}& (29)\end{array}$

where R_tr is the tracking result, represented by the estimated image region occupied by the tracked object, R_gt is the ground truth bounding box. R_tr∩R_gt is the intersection and R_tr∪R_gt is the union of the two regions. The range of the relative overlap is from 0 to 100%.

6.1. Results for the iEMD Tracker With Sparse Coding Histograms

In this subsection, the performance of the iEMD tracker with sparse coding histograms and the template update method is evaluated using the eight sequences. In our approach, the object windows are re-sized to 32 × 32 pixels for all the sequences, except for the Walking sequence, in which the object windows are resized to 64 × 32 pixels due to the smaller object size. The local patches in each object window are sampled with the size 16 × 16 pixels with step size 8 in sequences like Car4, Walking and Car2. For other sequences, the local patches in each object window are sampled with the size 8 × 8 pixels with step size 4. In the case of the abrupt motions of the object, 4 more particles are generated by moving the template in the surrounding area of the initial object position. For each particle, the template is enlarged and shrunk by 2% in case of the scale variations. Video 1 shows the comparisons of tracking methods on eight tracking video sequences (Supplementary Material section).

The performance of the proposed algorithm is compared with eight state-of-the-art tracking algorithms on eight video sequences. These state-of-the-art trackers include: ASLA (Jia et al., 2012), Frag (Adam et al., 2006), IVT (Ross et al., 2008), L1APG (Mei and Ling, 2011), LOT (Oron et al., 2012), MTT (Zhang et al., 2012), STRUCK (Hare et al., 2011), and iEMD(CH) (Yao et al., 2016; Yao and Dani, 2017). The source codes of the trackers are downloaded from the corresponding web pages and the default parameters are used. The average percentage overlap obtained by all the tracking algorithms on eight video sequences are reported in Table 2. The iEMD tracker achieves the highest average overlap over all the sequences. The iEMD tracker also achieves the second best tracking results on the 5 out of 8 sequences. In Figure 2, the tracking results of the eight video sequences are shown. The success plot shows the ratios of frames at the different thresholds of the relative overlap values varied from 0 to 1.

TABLE 2

Table 2. The average overlap (in percentage) obtained by the tracking algorithms on eight datasets.

FIGURE 2

Figure 2. Success plots (A–H) for the nine tracking algorithms on the eight sequences.

Representative tracking results obtained by iEMD algorithm are shown in Figure 3. In the Human8 and Bolt2 sequences, the targets have significant illumination variations, and deformations, respectively. Only LOT, iEMD and iEMD(CH) trackers are able to track targets in all the frames. The size of the template estimated by the iEMD(CH) tracker is shrinked and not accurate. The LOT and iEMD trackers use the EMD as the similarity measure and their appearance models are based on local image patches, which make the trackers more robust to illumination changes and deformations (Rubner et al., 2000; Oron et al., 2012). In woman sequence, all the trackers start to drift away from the target in frame 124 except for the iEMD and STRUCK trackers. For the Car2 and Car4 sequences, there are significant illumination changes when the targets pass underneath the trees and the overpasses. The LOT and Frag trackers start drifting away from frame 72 in Car2 sequences. In Car4 sequence, the LOT tracker starts to lose the target from frame 15, and the Frag and L1APG trackers drift away when the car passes the overpass in frame 249. In Walking2 sequence, the LOT, Frag, and ASLA trackers start tracking the wrong target in frame 246, due to the similar colors of the clothes between the two people.

FIGURE 3

Figure 3. The visual tracking results obtained by the nine tracking algorithms on the eight video sequences (No permission is required from the copyright holders and/or the depicted individuals for the use of these images. The original images are obtained from the CVPR 2013 database: http://cvlab.hanyang.ac.kr/tracker_benchmark/datasets.html).

6.2. Results for The Gyro-Aided iEMD Tracking Algorithm

The test of the gyro-aided iEMD tracking algorithm is conducted using the sequence including 100 frames from the dataset provided by CMU (Hwangbo et al., 2011). The images are taken in front of a desk with motions, such as shaking and rotation. The frame sequences have a resolution of 640 × 480 at 30 frames per second (FPS). The gyroscope is carefully aligned with the camera and the tri-axial gyroscopic values are sampled at 11Hz in the range of ±200deg/sec (Hwangbo et al., 2011). Using the time stamps of the camera and the gyroscope, the angular rate data are synchronized with the frames captured by the camera.

The comparisons between the tracking results using the iEMD tracker with and without the gyroscope information are illustrated in Figure 4. The head of the eagle is chosen as the target and the ground truth is manually labeled in each frame. The magenta box indicates the estimated image region without using the gyroscope data, and the cyan box is the tracking results of the gyro-aided iEMD tracker. Without the gyroscope data, the tracker loses the target after the frame 15. However, the head of the eagle is successfully tracked with our gyro-aided iEMD tracking algorithm.

FIGURE 4

Figure 4. Results of the iEMD tracker in presence of rapid camera motion; the magenta boxes indicate the results of the iEMD tracker without the gyroscope information, and the cyan boxes indicate the results of the gyro-aided iEMD tracker (No permission is required from the copyright holders and/or the depicted individuals for the use of these images. The original images are obtained from the database: http://www.cs.cmu.edu/~myung/IMU_KLT/).

The performances of the iEMD tracker with and without the gyroscope information on the CMU sequence are summarized in Table 3. The value of the average overlap, the percentage of the total frame numbers of which the overlap is greater than 0 and 40% are reported. Gyroscope information provides a good initial position for the iEMD tracker to estimate the location of the target. Thus, the gyro-aided iEMD tracking algorithm is robust to the rapid movements of the camera.

TABLE 3

Table 3. Evaluation results on the CMU dataset using the iEMD tracker with and without the gyroscope information.

7. Discussion

As a cross-bin metric for the comparison of the histograms, the advantages of the EMD are demonstrated in situations such as illumination variation, object deformation and partial occlusion. The iEMD algorithm uses the transportation-simplex method for calculating the EMD. The practical running time complexity of the transportation-simplex method is supercubic [a complexity in Ω(N³)⋂O(N⁴)] (Rubner et al., 2000), where N represents the number of the histogram bins. Other algorithms for calculating the EMD can be used to further reduce the running time (Ling and Okada, 2007; Pele and Werman, 2009). For our current impelementation in MATLAB, the average run time computed over the eight test sequences is 1.4 FPS. Compared to the algorithms used in Table 2, which has an average run time of 1 FPS or less (for algorithms implemented in MATLAB), iEMD algorithm performs better in terms of FPS (Wu et al., 2013). Furthermore, the experimental results, especially the Human8 and Bolt2 sequences, show that the iEMD tracker is robust to the appearance variations. The experimental results of Walking2 show that the iEMD tracker can discriminate the target from the surroundings with similar colors. The tracking results from Woman and Subway sequences demonstrate the robustness to partial occlusions. In Figures 2, 3 and Table 2, the tracking results of the iEMD(CH) (Yao et al., 2016; Yao and Dani, 2017) are also presented using six out of eight videos which has color images. The iEMD(CH) algorithm cannot be tested on Car2 and Car4 videos because they have gray images. Since the sparse coding histogram is used as the appearance model and the template update method is adopted to handle the appearance changes of the target, the performance of the iEMD with the sparse coding histogram is significantly better than our prior work using the iEMD(CH).

8. Conclusion and Future Work

This paper presents iEMD and gyro-aided iEMD visual tracking algorithms. The local sparse representation is used as the appearance model for the iEMD tracker. The maximum-alignment-pooling method is used for constructing a sparse coding histogram which reduces the computational complexity of the EMD optimization. The template update algorithm based on the EMD is also presented. The iEMD tracker is robust to variations in appearance of the target, deformations and partial occlusions. Experiments conducted on eight publicly available datasets show that the iEMD tracker is robust to the illumination changes, deformations and partial occlusions of the target. To validate the gyro-aided iEMD tracking algorithm, experimental results from the CMU dataset, which contains rapid camera motion are presented. Without the gyroscope measurements, the iEMD tracker fails on the CMU dataset. With the help of the gyroscope measurements, the iEMD algorithm is able to lock onto the target and track it successfully. The above experimental results show that the proposed iEMD tracking algorithm is robust to the appearance changes of the target as well as the ego-motion of the camera. As a gradient descent based dynamic model, the iEMD tracker, which provides good location prediction, can be further improved with more effective particle filters. The metrics used by sparse coding, such as the largest sum of the sparse coefficients or the smallest reconstruction error, can be combined with the EMD to make the tracker more discriminant. In future, an efficient impelementation of iEMD tracker in C/C++ will be pursued.

Author Contributions

GY: co-developed the algorithm and coded and validated the results. AD: co-developed the algorithm and verified the results.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

The authors would like to thank Prof. Peter Willett, Iman Salehi, and Harish Ravichandar for their help.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/frobt.2018.00095/full#supplementary-material

Video 1. Comparisons of Tracking Methods.

References

Adam, A., Rivlin, E., and Shimshoni, I. (2006). “Robust fragments-based tracking using the integral histogram,” in IEEE Computer Society Conf. on Computer Vision and Pattern Recognition (New York, NY), 798–805.

Google Scholar

Avidan, S. (2007). Ensemble tracking. IEEE Trans. Pattern Anal. Mach. Intell. 29, 261–271. doi: 10.1109/TPAMI.2007.35

PubMed Abstract | CrossRef Full Text | Google Scholar

Babenko, B., Yang, M. H., and Belongie, S. (2011). Robust object tracking with online multiple instance learning. IEEE Trans. Pattern Anal. Mach. Intell. 33, 1619–1632. doi: 10.1109/TPAMI.2010.226

PubMed Abstract | CrossRef Full Text | Google Scholar

Baum, M., Willett, P., and Hanebeck, U. D. (2015). On Wasserstein Barycenters and MMOSPA estimation. IEEE Signal Process. Lett. 22, 1511–1515. doi: 10.1109/LSP.2015.2410217

CrossRef Full Text | Google Scholar

Bolme, D. S., Beveridge, J. R., Draper, B. A., and Lui, Y. M. (2010). “Visual object tracking using adaptive correlation filters,” in IEEE Conference on Computer Vision and Pattern Recognition (San Francisco, CA), 2544–2550.

Google Scholar

Chwa, D., Dani, A. P., and Dixon, W. E. (2016). Range and motion estimation of a monocular camera using static and moving objects. IEEE Trans. Control Syst. Technol. 24, 1–10. doi: 10.1109/TCST.2015.2508001

CrossRef Full Text | Google Scholar

Collins, R. T., Liu, Y., and Leordeanu, M. (2005). Online selection of discriminative tracking features. IEEE Trans. Pattern Anal. Mach. Intell. 27, 1631–1643. doi: 10.1109/TPAMI.2005.205

PubMed Abstract | CrossRef Full Text | Google Scholar

Comaniciu, D., Ramesh, V., and Meer, P. (2003). Kernel-based object tracking. IEEE Trans. Pattern Anal. Mach. Intell. 25, 564–577. doi: 10.1109/TPAMI.2003.1195991

CrossRef Full Text | Google Scholar

Comport, A. I., Marchand, E., Pressigout, M., and Chaumette, F. (2006). Real-time markerless tracking for augmented reality: the virtual visual servoing framework. IEEE Trans. Visual. Comput. Graph. 12, 615–628. doi: 10.1109/TVCG.2006.78

PubMed Abstract | CrossRef Full Text | Google Scholar

Dani, A. P., Fischer, N. R., and Dixon, W. E. (2012). Single camera structure and motion. IEEE Trans. Automatic Control 57, 238–243. doi: 10.1109/TAC.2011.2162890

CrossRef Full Text | Google Scholar

Dani, A. P., Panahandeh, G., Chung, S.-J., and Hutchinson, S. (2013). “Image moments for higher-level feature based navigation,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (Tokyo), 602–609.

Google Scholar

Dantzig, G. B., and Thapa, M. N. (2006). Linear Programming 1: Introduction. Berlin: Heidelberg; Springer-Verlag.

Google Scholar

Davison, A. J., Reid, I. D., Molton, N. D., and Stasse, O. (2007). MonoSLAM: Real-time single camera SLAM. IEEE Trans. Pattern Anal. Mach. Intell. 29, 1052–1067. doi: 10.1109/TPAMI.2007.1049

PubMed Abstract | CrossRef Full Text | Google Scholar

Grabner, H., and Bischof, H. (2006). “On-line boosting and vision,” in IEEE Computer Society Conference on Computer Vision and Pattern Recognition (New York, NY), 260–267.

Google Scholar

Guerriero, M., Svensson, L., Svensson, D., and Willett, P. (2010). “Shooting two birds with two bullets: how to find minimum mean OSPA estimates,” in International Conference on Information Fusion (Edinburgh), 1–8.

Google Scholar

Hare, S., Saffari, A., and Torr, P. H. (2011). “STRUCK: structured output tracking with kernels,” in International Conference on Computer Vision (Barcelona), 263–270.

Hwangbo, M., Kim, J.-S., and Kanade, T. (2011). Gyro-aided feature tracking for a moving camera: fusion, auto-calibration and GPU implementation. Intl J. Robot. Res. 30, 1755–1774. doi: 10.1177/0278364911416391

CrossRef Full Text | Google Scholar

Jia, X., Lu, H., and Yang, M.-H. (2012). “Visual tracking via adaptive structural local sparse appearance model,” in IEEE Conference on Computer Vision and Pattern Recognition (Providence, RI), 1822–1829.

Google Scholar

Jia, X., Lu, H., and Yang, M. H. (2016). Visual tracking via coarse and fine structural local sparse appearance models. IEEE Trans. Image Process. 25, 4555–4564. doi: 10.1109/TIP.2016.2592701

PubMed Abstract | CrossRef Full Text | Google Scholar

Kalal, Z., Mikolajczyk, K., and Matas, J. (2012). Tracking-learning-detection. IEEE Trans. Pattern Anal. Mach. Intell. 34, 1409–1422. doi: 10.1109/TPAMI.2011.239

PubMed Abstract | CrossRef Full Text | Google Scholar

Karavasilis, V., Nikou, C., and Likas, A. (2011). Visual tracking using the earth mover's distance between gaussian mixtures and Kalman filtering. Image Vision Comput. 29, 295–305. doi: 10.1016/j.imavis.2010.12.002

CrossRef Full Text | Google Scholar

Li, H., Li, Y., and Porikli, F. (2016). Convolutional neural net bagging for online visual tracking. Comp. Vision Image Understand. 153, 120–129. doi: 10.1016/j.cviu.2016.07.002

CrossRef Full Text | Google Scholar

Ling, H., and Okada, K. (2007). An efficient earth mover's distance algorithm for robust histogram comparison. IEEE Trans. Pattern Anal. Mach. Intell. 29, 840–853. doi: 10.1109/TPAMI.2007.1058

PubMed Abstract | CrossRef Full Text | Google Scholar

Liu, B., Huang, J., Yang, L., and Kulikowsk, C. (2011). “Robust tracking using local sparse appearance model and K-selection,” in IEEE Conference on Computer Vision and Pattern Recognition (Colorado Springs, CO), 1313–1320.

Google Scholar

Mairal, J., Bach, F., and Ponce, J. (2014). Sparse modeling for image and vision processing. Foundations Trends Comput. Graph. Vis. 8, 85–283. doi: 10.1561/0600000058

CrossRef Full Text | Google Scholar

Marchand, É., and Chaumette, F. (2005). Feature tracking for visual servoing purposes. Robot. Auton. Syst. 52, 53–70. doi: 10.1016/j.robot.2005.03.009

CrossRef Full Text | Google Scholar

Marchand, É., Spindler, F., and Chaumette, F. (2005). ViSP for visual servoing: a generic software platform with a wide class of robot control skills. IEEE Robot. Automation Mag. 12, 40–52. doi: 10.1109/MRA.2005.1577023

CrossRef Full Text | Google Scholar

Mei, X., and Ling, H. (2009). “Robust visual tracking using L1 minimization,” in IEEE 12th International Conference on Computer Vision (Kyoto), 1436–1443.

Mei, X., and Ling, H. (2011). Robust visual tracking and vehicle classification via sparse representation. IEEE Trans. Pattern Anal. Mach. Intell. 33, 2259–2272. doi: 10.1109/TPAMI.2011.66

PubMed Abstract | CrossRef Full Text | Google Scholar

Oron, S., Bar-Hillel, A., Levi, D., and Avidan, S. (2012). “Locally orderless tracking,” in IEEE Conference on Computer Vision and Pattern Recognition (Providence, RI), 1940–1947.

Google Scholar

Park, J., Hwang, W., Kwon, H., and Kim, K. (2013). A novel line of sight control system for a robot vision tracking system, using vision feedback and motion-disturbance feedforward compensation. Robotica 31, 99–112. doi: 10.1017/S0263574712000124

CrossRef Full Text | Google Scholar

Pele, O., and Werman, M. (2009). “Fast and robust earth mover's distances,” in IEEE International Conference on Computer Vision, 460–467.

Google Scholar

Ravichandar, H. C., and Dani, A. P. (2014). “Gyro-aided image-based tracking using mutual information optimization and user inputs,” in IEEE International Conference on Systems, Man and Cybernetics (San Diego, CA), 858–863.

Google Scholar

Ravichandar, H. C., and Dani, A. P. (2015). “Human intention inference through interacting multiple model filtering,” in IEEE Conference on Multisensor Fusion and Integration (San Diego, CA).

Google Scholar

Ross, D. A., Lim, J., Lin, R.-S., and Yang, M.-H. (2008). Incremental learning for robust visual tracking. Intl. J. Comput. Vis. 77, 125–141. doi: 10.1007/s11263-007-0075-7

CrossRef Full Text | Google Scholar

Rubner, Y., Tomasi, C., and Guibas, L. J. (2000). The earth mover's distance as a metric for image retrieval. Intl J. Comput. Vis. 40, 99–121. doi: 10.1023/A:1026543900054

CrossRef Full Text | Google Scholar

Santner, J., Leistner, C., Saffari, A., Pock, T., and Bischof, H. (2010). “Prost: parallel robust online simple tracking,” in IEEE Conference on Computer Vision and Pattern Recognition (San Francisco, CA), 723–730.

Google Scholar

Spong, M. W., Hutchinson, S., and Vidyasagar, M. (2006). Robot Modeling and Control, Vol 3. New York, NY: Wiley.

Google Scholar

Tahri, O., Usman, M., Demonceaux, C., Fofi, D., and Hittawe, M. (2016). “Fast earth mover's distance computation for catadioptric image sequences,” in IEEE International Conference on Image Processing (Phoenix, AZ), 2485–2489.

Google Scholar

Vojir, T., Matas, J., and Noskova, J. (2016). Online adaptive hidden markov model for multi-tracker fusion. Comput. Vis. Image Understand. 153, 109–119. doi: 10.1016/j.cviu.2016.05.007

CrossRef Full Text | Google Scholar

Wang, J., Wang, H., and Zhao, W.-L. (2015). Affine hull based target representation for visual tracking. J. Vis. Commun. Image Represent. 30, 266–276. doi: 10.1016/j.jvcir.2015.04.014

CrossRef Full Text | Google Scholar

Wang, J., Wang, Y., and Wang, H. (2016). “Adaptive appearance modeling with Point-to-Set metric learning for visual tracking,” in IEEE Transaction on Circuits and Systems for Video Technology. doi: 10.1109/TCSVT.2016.2556438

CrossRef Full Text | Google Scholar

Wu, Y., Lim, J., and Yang, M.-H. (2013). “Online object tracking: a benchmark,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (Portland, OR).

Google Scholar

Yang, J., Dani, A. P., Chung, S.-J., and Hutchinson, S. (2015). Vision-based localization and robot-centric mapping in riverine environments. J. Field Robot. 34, 429–450. doi: 10.1002/rob.21606

CrossRef Full Text | Google Scholar

Yao, G., and Dani, A. (2017). Gyro-aided visual tracking using iterative Earth mover's distance. IEEE Aerospace Electron. Syst. Mag. 32, 52–55. doi: 10.1109/MAES.2017.160223

CrossRef Full Text

Yao, G., Williams, M., and Dani, A. (2016). “Gyro-aided visual tracking using iterative Earth mover's distance,” in 19th International Conference on Information Fusion (FUSION) (Heidelberg), 2317–2323.

Google Scholar

Yilmaz, A., Javed, O., and Shah, M. (2006). Object tracking: a survey. ACM Comput. Surveys 38:13. doi: 10.1145/1177352.1177355

CrossRef Full Text | Google Scholar

Zhang, S., Yao, H., Sun, X., and Lu, X. (2013). Sparse coding based visual tracking: review and experimental comparison. Pattern Recogn. 46, 1772–1788. doi: 10.1016/j.patcog.2012.10.006

CrossRef Full Text | Google Scholar

Zhang, T., Ghanem, B., Liu, S., and Ahuja, N. (2012). “Robust visual tracking via multi-task sparse learning,” in IEEE Conference on Computer Vision and Pattern Recognition (Providence, RI), 2042–2049.

Google Scholar

Zhang, Z., and Hong Wong, K. (2014). “Pyramid-based visual tracking using sparsity represented mean transform,” in Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (Columbus, OH), 1226–1233.

Google Scholar

Zhao, Q., Yang, Z., and Tao, H. (2010). Differential earth mover's distance with its applications to visual tracking. IEEE Trans. Pattern Anal. Mach. Intell. 32, 274–287. doi: 10.1109/TPAMI.2008.299

PubMed Abstract | CrossRef Full Text | Google Scholar

Zhu, G., Wang, J., and Lu, H. (2016). Clustering based ensemble correlation tracking. Comp. Vis. Image Understand. 153, 55–63. doi: 10.1016/j.cviu.2016.05.006

CrossRef Full Text | Google Scholar

Zivkovic, Z., Cemgil, A. T., and Kröse, B. (2009). Approximate Bayesian methods for Kernel-based object tracking. Comp. Vis. Image Understand. 113, 743–749. doi: 10.1016/j.cviu.2008.12.008

CrossRef Full Text | Google Scholar