This paper presents a phonetic segment classification method based on joint embeddings learned from processing Ultrasound Tongue Imaging (UTI) and audio data. For constructing the embeddings, we compiled an ultrasound image dataset synchronized with audio that encompasses common speech scenarios. The embeddings are obtained from artificial neural network models trained on this dataset. During testing, our model processes only audio data, making it practical for speech therapy as no ultrasound imaging is required. Experiments show that our method yields similar performance compared to methods that simultaneously use both audio and UTI data. However, it outperforms the methods utilizing solely audio or UTI data in real-time classification.

It is well known in the machine learning community that the ensembles of neural networks outperform the respective individual networks; hence recent work on aerial scene classification has been trending toward various network fusion strategies. However, training multiple deep networks can be computationally expensive. “Snapshot ensembling” has recently appeared as an alternative and claims to provide the performance of network ensembles at the cost of a single network's training. In this letter, we present the results of a comparative study on deep network ensembles in the context of aerial scene classification, where homogeneous, heterogeneous, and snapshot ensembling strategies are explored with both DenseNet and Inception networks; contrary to the existing work, we employ model fusion at the last convolutional layer's level. The explored approaches are validated with the two largest and most challenging data sets available (AID and NWPU-RESISC45), and state-of-the-art results are achieved on both.

In this paper we propose an exploratory augmented reality(AR) system for mobile devices. A hybrid system of 3D object detection and 3D tracking is used to rapidly localize the object in the scene. Based on randomized tree classifier, object detection supports large viewpoint changes. While edge-based 3D tracking provides efficient computation and better accuracy for pose estimation of incremental motions. The result is an augmented reality system that works well with large viewpoint variance and has superior accuracy. Using a mobile device such as iPad or iPhone, our system further provides exploratory capability using their touch screen and wireless connectivity. The user is able to interact and explore the 3D object on the image or video, and collaborate with a remote user. Extensive experiments with different subjects demonstrate that the proposed system advances the state-of-the-art in augmented reality with novel and intuitive applications.

Traditional short-range wireless network standards, such as IEEE 802.11 and IEEE 802.15.4, do not consider packets according to their delay deadlines and they treat the packets all the same regardless of these requirements. In this document, we propose a cross-layer architecture where the application using the wireless medium associates each packet it is pushing down the communication stack with a delay requirement. We propose two new mechanisms: ER (Early Retirement) and PHY+. These mechanisms utilize the deadline associated with each packet for further transmission and redundancy optimization. In our simulations, we focus on the Wireless Sensor Networks domain and apply our methods as an extension to the well-known and widely acceptable low-power, low-throughput IEEE 802.15.4 standard. Simulations performed showed that our mechanisms outperform the original standard considerably, especially in situations where high numbers of users exist in the system.

We propose a new edge detection algorithm that works by computing a set of anchor edge points in an image and then linking these anchor points by drawing edges between them. The resulting edge map consists of perfect contiguous, one pixel wide edges. The performance tests show that our algorithm is up to 16% faster than the fastest known edge detection algorithm, i.e., OpenCV implementation of the Canny edge detector. We believe that our edge detector is a novel step in edge detection and would be very suitable for the next generation real-time image processing and computer vision applications.

Vehicles are indispensable in modern life. The capability of monitoring them over a long range can play significant roles in many surveillance applications. However, due to high mobility of vehicles, tracking them is difficult and we need to utilize a large network of cameras and reason on discrete sets of observations made from non-overlapping cameras. In this paper, we introduce enabling techniques for such a surveillance need. Specifically, we build explicit 3D models and use them for vehicle signature extraction and matching. The algorithm uses a single active shape model (ASM) for all consumer vehicles. After detecting presence of a vehicle, \eg, by background subtraction, our algorithm then reconstructs a texture mapped 3D model. 3D car models enable us to monitor vehicles in many novel ways otherwise impossible. Two use cases are provided.

Analytic manifolds were recently used for motion averaging, segmentation and robust estimation. Here we consider the epipolar constraint for calibrated cameras, which is the most general motion model for calibrated cameras and is encoded by the essential matrix. The set of all essential matrices forms the essential manifold. We provide a theoretical characterization of the geometry of the essential manifold and develop a parametrization which associates each essential matrix with a unique point on the manifold. Our work provides a more complete theoretical analysis of the essential manifold than previous work in this direction. We show the results of using this parametrization with real data sets, while previous work concentrated on theoretical analysis with synthetic data.

We present a novel feature representation for categorical object detection. Unlike previous approaches that have concentrated on generic interest-point detectors, we construct object-specific features directly from the training images. Our feature is represented by a collection of Flexible Edge Arrangement Templates (FEATs). We propose a two-stage semi-supervised learning approach to feature selection. A subset of frequent templates are first selected from a large template pool. In the second stage, we formulate feature selection as a regression problem and use LASSO method to find the most discriminative templates from the preselected ones. FEATs adaptively capture the image structure and naturally accommodate local shape variations. We show that this feature can be complemented by the traditional holistic patch method, thus achieving both efficiency and accuracy. We evaluate our method on three well-known car datasets, showing performance competitive with existing methods.

Model-based 3D object tracking is fast and robust using 3D edges. However, traditional edge-based approaches have difficulty handling occluding contours of curved surfaces, since they are not static model edges but change with the viewpoint. In this paper we propose a unified approach to edge-based tracking where 3D edges including occluding contours are utilized. This is achieved through an analysis of local surface differential geometry, which provides the foundation for incorporating occluding contours of curved surfaces into edge-based tracking. This approach uses a simple parametrization of both types of model edges within the same framework. The proposed method has been tested within the context of an existing edge-based tracking system. The system can track both types of model edges in a very fast and robust manner. Experimental results on both synthetic and real scenes are provided, which confirm that occluding contours improve real-time 3D tracking performance.

Reliability of a model-based edge tracker critically depends on its ability to establish correct correspondences between points on the model edges and edge pixels in an image. This is a non-trivial problem especially in the presence of large inter-frame motions and in cluttered environments. We propose an online learning approach to solving this problem. An edge pixel is represented by a descriptor composed of a small segment of intensity patterns. From training examples the algorithm utilizes the randomized forest model to learn a posteriori distribution of correspondence given the descriptor. In a new frame, the edge pixels are classified using maximum a posteriori (MAP) estimation. The proposed method is very powerful and it enables us to apply the proposed tracker to many previously impossible scenarios with unprecedented robustness.

We propose a new robust estimator for camera pose estimation based on a recently developed nonlinear mean shift algorithm. This allows us to treat pose estimation as a clustering problem in the presence of outliers. We compare our method to RANSAC, which is the standard robust estimator for computer vision problems. We also show that under fairly general assumptions our method is provably better than RANSAC. Synthetic and real examples to support our claims are provided

This paper presents a method for real-time wide-baseline feature matching. The approach is based on the work of Lepetit and colleagues [9], where randomized decision trees are trained to establish correspondences between detected features in a training image, and those in input frames. Though extremely promising, their actual results can vary depending on the viewpoint and illumination conditions. We combine two approaches to alleviate its limitations. The first aims to update the trees at run-time, adapting them to the actual viewing conditions. The second consists in spatially distributing the trees, so that each of them models a certain viewing volume more precisely. The result is a more stable matching method that significantly extends detectable range and is much more robust to illumination changes, such as cast shadows or reflections. 1

This paper deals with part-based object recognition. We present a statistical shape model that characterizes both variations intrinsic to an object class and perturbations unique to each observation. Object parts, constrained by the learned shape model, are efficiently localized using the max-product algorithm on a triangulated Markov random field (TMRF). As a combination of both techniques, our system is able to recognize objects with large deformations and locate their parts accurately. We test our method on standard databases. The recognition performance is compared to the state of the art and favorable results are reported. 1

Object detection is challenging partly due to the limited discriminative power of local feature descriptors. We amend this limitation by incorporating spatial constraints among neighboring features. We propose a two-step algorithm. First, a feature together with its spatial neighbors forms a flexible feature template. Two feature templates can be compared more informatively than two individual features without knowing the 3D object model. A large portion of false matches can be excluded after the first step. In a second global matching step, object detection is formulated as a graph-matching problem. A model graph is constructed by applying Delaunay triangulation on the surviving features. The best matching graph in an input image is computed by finding the maximum a posterior (MAP) estimate of a binary Markov random field with triangular maximal clique. The optimization is solved by the max-product algorithm (a.k.a. belief propagation). Experiments on both rigid and non-rigid objects demonstrate the generality and efficacy of the proposed methods.

Object detection is challenging partly due to the limited discriminative power of local feature descriptors. We propose a two-step algorithm. First, a feature together with its spatial neighbors forms a flexible feature template. Two feature templates can be compared more informatively than two individual features without knowing the 3D object model. A large portion of false matches can be excluded after the first step. In a second global matching step, object detection is formulated as a graph-matching problem. A model graph is constructed by applying Delaunay triangulation on the surviving features. The best matching graph in an input image is computed by finding the maximum a posterior (MAP) estimate of a binary Markov random field with triangular maximal clique. The optimization is solved by the max-product algorithm (a.k.a. belief propagation). Experiments on both rigid and nonrigid objects demonstrate the generality and efficacy of the proposed methods.

Estimation of camera pose is an integral part of augmented reality systems. Vision-based methods offer a flexible and accurate method for this estimation. Current vision based methods rely on markers to reduce the computation and increase robustness of the pose estimation. However, this limits the algorithm's applicability while being expensive since the markers also require maintenance. Alternatively, reconstructed scene features can be used for pose estimation but this can lead to a loss of accuracy. To avoid this we propose a two-stage balanced tracking method which does not require any visual markers in the scene. The first stage of our method is based on the sequential recovery of structure from motion which allows the system to learn the scene from a few frames in which the markers are visible. In the next stage, the learned features are used for camera tracking. The system ensures greater accuracy and reduces error drift due to its use of the HEIV estimator which is provably unbiased to the first degree. We also make use of a novel method for the detection and removal of outliers which are unavoidable in such systems. The experiments show the superiority of our method when compared to a nonlinear method based on Levenberg-Marquardt minimization.

This paper introduces a "diminished reality" technique for removing an object or collection of objects and replacing it with an appropriate background image. Diminished reality can be considered an important part of many mixed and augmented reality applications. Our target application is the use of augmented reality (AR) to revamp procedures in industrial plants. An object or a region of interest is delineated on a single reference image. A paraperspective projection model is used to generate the correct background from multiple calibrated views of the scene. We propose methods to deal with approximately planar backgrounds with different orientations. We also propose a multi-resolution approach to deal with non-planar backgrounds. Different sets of experimental results demonstrate the success and limits of the algorithms. Results on real data from water treatment and power plants show the usefulness of this method for industrial applications.

We introduce a linear algorithm to recover the Euclidean motion between an orthographic and two perspective cameras from straight line correspondences filling the gap in the analysis of motion estimation from line correspondences for various projection models. The general relationship between lines in three views is described by the trifocal tensor. Euclidean structure from motion for three perspective views is a special case in which the relationship is defined by a collection of three matrices. Here, we describe the case of two calibrated perspective views and an orthographic view. Similar to the other cases, our linear algorithm requires 13 or more line correspondences to recover 27 coefficients of the trifocal tensor.

Estimating the pose of a camera (virtual or real) in which some augmentation takes place is one of the most important parts of an augmented reality (AR) system. The availability of powerful processors and fast frame grabbers has made the use of vision-based trackers commonplace due to their accuracy as well as flexibility and ease of use. Current vision-based trackers are based on tracking of markers. The use of markers increases robustness and reduces computational requirements. However, their use can be very complicated, as they require maintenance. Direct use of scene features for tracking, therefore, is desirable. To this end, we describe a general system that tracks the position and orientation of a camera observing a scene without visual markers. Our method is based on a two-stage process. In the first stage, a set of features is learned with the help of an external tracking system during use. The second stage uses these learned features for camera tracking when the system in the first stage decides that it is possible to do so. The system is very general so that it can employ any available feature tracking and pose estimation system for learning and tracking. We experimentally demonstrate the viability of the method in real-life examples.

Registration is a crucial task in a see-through augmented reality (AR) system. The importance stems not only from the fact that registration requires careful calibration but also from the necessity that any calibration procedure should take users into account. Tuceryan et al. (2002) proposed a general method for calibrating a see-through device based on dynamic alignment of virtual and real points. Although a powerful tool, our experiments showed that users find alignment of many points overwhelming. We introduce improvements to simplify the calibration process and increase the success rate. We first identified why calibration parameters differ from user to user and how this can be prevented by adopting particular configurations for the tracker sensor and display. This allowed us to re-use the existing calibrations. Furthermore, we have introduced a simpler model for the calibration that requires fewer user inputs, typically four, to calibrate the system.

Augmented reality (AR) is a technology that enhances the user's view of real world with virtual objects. AR is a relatively new field and seeks applications in a wide range of areas from medicine to industry. Applying AR with mobile computing, we present a new method for information access in industrial environments. We, in this paper, present a framework that is applicable to different scenarios in larger industrial settings for spatial data access, navigation, and image augmentation in realtime. In the core of our framework lies a mobile computer equipped with a camera. The camera observes the environment for visual coded markers. These markers are registered to a global coordinate system through available drawings or floor plans. The tracker software processes images coming from the camera at moderate frame rates (typically 12fps) and estimates the location of the user. The system then can guide the user through the environment, provide the user with location relevant data from a spatial database, and augment the view of the user through the camera. Data exchange is done via a wireless network and the user interface allows the user to access various types of data without considerable effort.

We describe algorithms for computing projective structure and motion from a multi-image sequence of tracked points. The algorithms are essentially linear, work for any motion of moderate size, and give accuracies similar to those of a maximum-likelihood estimate. They give better results than the factorization approach of Sturm and Triggs (1996) and are equally fast and they are much faster than bundle adjustment. Our experiments show that the (iterated) Sturm-Triggs approach often fails for linear camera motions. In addition, we study experimentally the common situation where the calibration is fixed and approximately known, comparing the projective versions of our algorithms to mixed projective/Euclidean strategies. We clarify the nature of dominant-plane compensation, showing that it can be considered a small-translation approximation rather than an approximation that the scene is planar. We show that projective algorithms accurately recover the (projected) inverse depths and homographies despite the possibility of transforming the structure and motion by a projective transformation.

In this paper we introduce a linear algorithm to recover the Euclidean motion between one orthographic and two perspective cameras from straight line correspondences filling the gap in the analysis of motion estimation from line correspondences for various projection models. The general relationship between lines in three views is described by the trifocal tensor. Euclidean structure from motion for three perspective views is a special case in which the relationship is defined by a collection of three matrices. We describe the case of two calibrated perspective views and an orthographic view. Similar to the other cases, our linear algorithm requires 13 or more line correspondences to recover 27 coefficients of the trifocal tensor.

This paper addresses the problem of constructing object models from various types of images. After a brief discussion of current approaches to this problem, we focus on two of its instances: the construction of three-dimensional surface models from object outlines found in a small set of registered photographs; and the synthesis of new images of a scene without any explicit three-dimensional reconstruction (image-based rendering). In both cases, we discuss the state of the art and present some of our recent work as an illustration of what can be achieved today.

This paper addresses the problem of reliably estimating the coefficients of the parameterized image variety (PIV) associated with the set of weak perspective images of a rigid scene, with applications in image-based rendering. Exploiting the fact that the constraints defining the PIV are linear in its coefficients and bilinear in the image data, the estimation procedure is cast in the errors-in-variables framework and solved using the method proposed by Y. Leedan and P. Meer (1998) for this type of problems. The proposed approach has been implemented, and experiments with real data are shown to yield much better prediction power than the original method based on singular value decomposition. Extensions to the more difficult case of paraperspective projection are briefly discussed.

We describe new algorithms for multi-frame structure from motion from tracked point features. The algorithms are essentially linear and give accuracies similar to those of a maximum-likelihood estimate. For the common situations where the calibration is fixed and approximately known, we experimentally compare the fully projective versions of our algorithms to mixed projective/Euclidean strategies. Our theoretical results clarify the nature of dominant-plane compensation and the effect of calibration error on translation recovery.

We describe two new algorithms for two-frame structure from motion from tracked point features. One is the first fast algorithm for computing an exact least-squares estimate. It exploits our observation that the rotationally invariant least-squares error can be written in a simple form that depends just on the motion. The other is essentially as accurate as the least-squares estimate and is more efficient, probably faster, and potentially more robust than previous algorithms of comparable accuracy. We also analyze theoretically the accuracy of the optical-flow approximation to the least-squares error.

This paper addresses the problem of characterizing the space formed by all images of a rigid set of n points observed by a weak perspective or paraperspective camera. By taking explicitly into account the Euclidean constraints associated with calibrated cameras, we show that this space is a six-dimensional variety embedded in R/sup 2n/, and parameterize it using the image positions of three reference points. This parameterization is constructed via linear least squares from point correspondences established across a sequence of images, and it is used to synthesize new pictures without any explicit three-dimensional model. Degenerate scene and camera configurations are analyzed, and experiments with real image sequences are presented.

This paper addresses the problem of estimating the epipolar geometry from point correspondences between two images taken by uncalibrated perspective cameras. It is shown that Jepson's and Heeger's linear subspace technique for infinitesimal motion estimation can be generalized to the finite motion case by choosing an appropriate basis for projective space. This yields a linear method for weak calibration. The proposed algorithm has been implemented and tested on both real and synthetic images, and it is compared to other linear and non-linear approaches to weak calibration.