An efficient architecture for image descriptor matching that uses a partitioned content-addressable memory (CAM)-based approach is proposed. CAM is frequently used in high-speed content-matching applications. However, due to its lack of functionality to support approximate matching, conventional CAM is not directly useful for image descriptor matching. Our modifications improve the CAM architecture to support approximate content matching for selecting image matches with local binary descriptors. Matches are based on Hamming distances computed for all possible pairs of binary descriptors extracted from two images. We demonstrate an FPGA-based implementation of our CAM-based descriptor-matching unit to illustrate the high matching speed of our design. The time complexity of our modified CAM method for binary descriptor matching is O(n). Our method performs binary descriptor matching at a rate of one descriptor per clock cycle at a frequency of 102 MHz. The resource utilization and timing metrics of several experiments are reported to demonstrate the efficacy and scalability of our design.

Image matching is a key component in computer vision and has many applications. In this work, we propose a circular shifting binary descriptor to increase the speed of rotation invariant image matching algorithms. We can compute the descriptors of the rotated image patches without rotating either the sample points or the image patch by circularly shifting the binary descriptor. Thus, operations such as multiplications and divisions from the orientation estimation step are eliminated from the image matching process which significantly reduces the number of operations for computing the descriptor. In addition, our experiments illustrate that the circular shifting binary descriptor shows limited rotation error in comparison with other descriptors such as ORB while attaining comparable mean average precision.

Binary descriptors have been shown to be faster than nonbinary descriptors while producing comparable results in image matching applications. In recent years, there have been many attempts to design hardware accelerators for extraction of binary descriptors to achieve higher processing rates. One of the well-known methods is the binary robust invariant scalable key point (BRISK) algorithm, which has shown outstanding results in various applications. In this work, we propose a multiscale field-programmable gate array (FPGA)-based hardware architecture for the BRISK descriptor. In addition, a new image sampling pattern for the BRISK algorithm is described which is shown to be more efficient than the original sampling pattern for hardware implementation. Our new sampling pattern decreases the size of the patches containing the key point to one-quarter of the size of that used in the original BRISK algorithm, which leads to a reduction in FPGA resource utilization while maintaining the accuracy of the image matching application. Our proposed design is fully pipelined and achieves a frame rate of 78 fps on images with full HD resolution.

This paper focuses on video summarization of abnormal behavior for remote invigilation of online exams. While the last decade has seen a massive increase in e-learning and online courses offered at postsecondary institutions, preserving the integrity of online examinations still heavily relies on web video conference invigilation performed by a remote proctor. Live remote invigilation is limited in the number of students that can be handled at once, and manual post-exam review is labor intensive. We propose a novel computer vision-based video content analysis system for the automatic creation of video summaries of online exams to assist remote proctors in post-exam reviews. The proposed method models normal and abnormal student behavior patterns using head pose estimations and a semantically meaningful two-state hidden Markov model. Video summaries are created from detected sequences of abnormal behavior. Experimental results are promising and demonstrate the viability of the proposed approach, which could readily be expanded to generate real-time alerts for live remote invigilation.

In this paper, we present a challenging dataset for the purpose of segmentation and change detection in photographic images of mountain habitats. We also propose a baseline algorithm for habitats segmentation to allow for performance comparison. The dataset consists of high resolution image pairs of historic and repeat photographs of mountain habitats acquired in the Canadian Rocky Mountains for ecological surveys. With a time lapse of 70 to 100 years between the acquisition of historic and repeat images, these photographs contain critical information about ecological change in the Rockies. The challenging aspects of analyzing these image pairs come mostly from the perspective (oblique) view of the photographs and the lack of color information in the historic photographs. The baseline algorithm that we propose here is based on texture analysis and machine learning techniques. Classifier training and results validation are made possible by the availability of expert manual ground-truth segmentation for each image. The results obtained with the baseline algorithm are promising and serve as a reference for new and improved segmentation and change detection algorithms.

A novel framework for acceleration of particle filtering approaches to 3D model-based, markerless visual tracking in monocular video is described. Specifically, we present a methodology for partitioning and mapping the computationally expensive weight-update stage of a particle filter to a graphics processing unit (GPU) to achieve particle- and pixel-level parallelism. Nvidia CUDA and Direct3D are employed to harness the massively parallel computational power of modern GPUs for simulation (3D model rendering) and evaluation (segmentation, feature extraction, and weight calculation) of hundreds of particles at high speeds. The proposed framework addresses the computational intensity that is intrinsic to all particle filter approaches, including those that have been modified to minimize the number of particles required for a particular task. Performance and tracking quality results for rigid object and articulated hand tracking experiments demonstrate markerless, model-based visual tracking on consumer-grade graphics hardware with pixel-level accuracy up to 95 percent at 60+ frames per second. The framework accelerates particle evaluation up to 49 times over a comparable CPU-only implementation, providing an increased particle count while maintaining real-time frame rates.

A correlation-based optical flow algorithm using compute unified device architecture (CUDA) technology to achieve fast motion-based image segmentation is described. Using CUDA, a 240 processor GPU implementation of an optimized correlation-based optical flow algorithm allows segmentation to be achieved at high frame rates on high-resolution video sequences. Details of the mapping of the optical flow segmentation algorithm onto the CUDA architecture as well as performance results are given. The performance of the algorithm is further characterized as a function of the search and correlation window radii.

In this paper, we propose an efficient mode decision algorithm for inter frame coding in H.264/AVC. From examination of the residue between the current and a reconstructed previous macroblock, the proposed algorithm selects modes before calculating the cost function. When the proposed algorithm is adopted in H.264/AVC it is shown to have good performance. Also, the skip mode decision is done before calculating the cost function of the sub-macroblock mode. Experimental results show how the proposed algorithm reduces encoding and motion estimation time in comparison of the performance of the proposed algorithm with the conventional H.264/AVC.

High-fidelity haptic interaction in multi-user environments over general Ethernet-based local area networks (LAN) and metropolitan area networks (MAN) can be challenging but has promising applications. Under typical network traffic conditions, the 1kHz real-time control rate suggested in the literature for stable haptic simulation is well above that achievable by conventional network protocols such as the UDP and TCP/IP. To overcome this limitation, a decentralized multi-rate control approach is proposed in which local force-feedback loops are executed at higher rates than data packet transmission between the user workstations. Mathematical models for stability and performance analysis of such multi-rate haptic control systems are presented. Analytical and experimental results demonstrate improved performance and stability for the distributed control architecture when compared with a centralized controller.

A method is presented for deriving an optimal basis for features classified with a support vector machine. The method is based on minimizing the leave-one-out error which is approximated by the radius-margin bound. A gradient descent method provides a learning rule for the basis in an outer loop of an iteration. The inner loop performs support vector machine training and provides support vector coefficients on which the gradient descent depends. In this way, the derivation of a basis for feature extraction and the support vector machine are jointly optimized. The efficacy of the method is illustrated with examples from multi-dimensional synthetic data sets

A novel system for face detection in images and video sequences is presented. The system incorporates a two-stage linear discriminant and nonlinear support vector machine classifier coupled with a front-end biologically-inspired search scheme. Results based on the CMU test set demonstrate that by using such a classifier arrangement with a non-exhaustive searching scheme, a significant reduction in computational complexity is achieved while maintaining comparable accuracy to other leading face detection systems.

A new, area-based mesh simplification algorithm is described. The proposed algorithm removes the center vertex of a polygon which consists of n/spl ges/3 faces and represents that polygon with n-2 faces. A global search method is introduced that iteratively determines which vertex is to be removed using the proposed area-based distortion measurement. Various re-triangulations are also considered to improve the perceptual quality of the final approximation. Experimental results demonstrate the performance of the proposed algorithm for data reduction while maintaining the quality of the rendered objects.

An error concealment algorithm is proposed based on flow of facial expression to improve communication of animated facial data over a limited bandwidth and error prone channel. Facial expression flow is tracked using dominant muscles which are those with maximum change between two successive frames. The receiver uses linear interpolation and the information on facial expression flow to interpolate the erroneous facial animation data. Experimental results are provided to show that the proposed error concealment method improves the quality of an animated face communications.

An algorithm that discovers a compact data representation for support vector classification is presented. The algorithm finds a basis which reduces the volume occupied by the coefficients in subspace. This volume reduction is driven by the support vectors of a support vector machine. A compact support vector representation (CSVR) of this form is shown to exhibit good generalization in the form of large margin and a small number of support vectors, while achieving low classification error rates. The compact nature of the data representation is shown to be particularly effective in representing correlated image sets such as those found in databases where faces and objects are imaged under varying lighting or pose.

A network of real-time vision sensors is described using 100baseT Ethernet. Each vision sensor comprises a camera connected to a dedicated computer to form a "vision node". Each vision node is capable of capturing and processing images and communicating the results over the network to a master computer in real-time. Deterministic and timely operation using standard Ethernet is accomplished using three basic techniques: suppressing superfluous packets; using the UDP transport protocol; and using an application layer protocol based on master/slave round-robin polling. Deterministic communications also enable the acquisition times of the cameras to be synchronized over the Ethernet network. By synchronizing the vision nodes to interleave their capture times, network contention is avoided and the effective video sample rate is increased. An analysis is also performed to determine the maximum number of vision nodes that can be supported using this technique over 100baseT Ethernet.

A direct visual servoing system is described which employs a network of cameras providing high-speed vision feedback that is robust to occlusions. This system does not rely on any external position or velocity sensors, but directly sets motor current using visual feedback alone. The limitation of 60 Hz video is overcome with multiple RS-170 cameras, synchronized over a network in round-robin fashion, to capture video fields at different instants in time. Each camera has its own computer that processes video at field rates to determine the position of a planar robot joint using eigenspace methods. The eigenspace computations produce position and Euclidean distance measurements sent from each camera node over a network to a master servo computer. It is shown that the Euclidean distance from the manifold in eigenspace in the presence of random occlusions is statistically related to the position measurement error. Occlusions are thus considered as "noise," and the measurement error variance is estimated directly from Euclidean distance. The measurement error variance is applied directly to a Kalman filter, which weights feedback from each camera to provide improved position estimates. The Kalman filter also models the vision transport delays to provide timely position estimates to ensure the stable direct visual servoing of a planar robot. Simulation results illustrate improvement in dynamic performance as the number of cameras are increased. Experimental measurements were obtained for a network of four cameras performing direct visual servoing of a simple planar robot. The results demonstrate the step response, as well as stable servo-hold operation in the presence of full occlusions in a subset of cameras or with partial occlusions in all cameras.

A method employing independent component analysis (ICA) to measure the position of a camera and the pose of an object from images in the presence of occlusion is presented. ICA is used to provide a low-dimensional representation of a set of images taken throughout a range of camera positions and object poses. Using the low-dimensional independent component subspace arbitrary camera locations or object poses can then be determined. The ICA technique is compared with principal component analysis (PCA) in the presence of varying degrees of occlusion. Occlusions of translation, pan and pose images were experimentally applied to provide a demonstration of the performance of this method. The independent component subspace is shown to provide more accurate position and pose information than PCA.

A distributed camera system using off-the-shelf components is presented that demonstrates the capability to perform high-speed vision feedback suitable for applications such as direct visual servoing. The limitation of 60 Hz video sample rates is overcome by using multiple RS-170 cameras synchronized over a network to capture at different instants in time. Each camera node has its own computer that processes video at field rates to determine the pose of a planar robot joint using eigenspace methods. Position information is fed back over a network to a master computer to perform direct visual servoing. The resulting vision feedback from the multiple cameras uses a Kalman filter to estimate position and to model the vision computation and transport delays. Computer simulation results are provided as the number of cameras are varied. Finally, real-time experimental results are presented that verify the approach using a network of four cameras performing direct visual servoing of a simple planar robot.

A multiple camera, visual servo system is presented that tracks and aligns rigid objects in 3D using wire-frame models. The system is capable of achieving frame-rate (30 Hz between two cameras) object tracking while using nonlinear iterative pose estimation. Tracking begins with a rough estimate of the positions of the objects. The relative rotation and translation of the cameras (external calibration) is calculated from the images. Six degree-of-freedom object pose estimates are computationally fused to provide tracking that is both stable and robust to occlusion. The tracking data is used to move (servo) the objects together while performing online robot Jacobian estimation. The robot is controlled in real time by joint angles with no prior knowledge of its kinematics.

An algorithm for generating skeletons of objects in a binary image is described. The algorithm produces a well-centered skeleton with the same simple connectivity as the object, and it allows the object to be either exactly or approximately (to within a known, user-selectable error) reconstructed. Its connectivity and reconstructability properties can be rigorously proved. For approximate reconstruction, the skeleton can also be (almost always) thin and is insensitive to border noise without image prefiltering or skeleton post-pruning, while maintaining the precise error bounds for reconstruction. Because of these properties, its robustness to rotation, pleasing visual appearance, and flexibility, it is well suited for such applications as data compression, image analysis, character recognition, and circuit board inspection.>

An algorithm for generating connected skeletons of objects in binary images is described. Three main properties of the algorithm are that: (1) it is noniterative, taking a fixed number of passes through the image to produce the skeleton regardless of the width of the objects; (2) it is based on a distance transform that uses a good approximation to the Euclidean distance, giving skeletons that are well centered and robust with respect to rotation; and (3) the skeletons it produces are connected. In addition, the skeletons are thin and allow the objects to be nearly reconstructed. The algorithm can also be run in a mode to produce centerlines, a connected approximation to the skeleton that is less sensitive to border noise and that is useful in image analysis applications.>

An experimental system for automatic visual measurement of surface-mount device placement on printed circuit boards has been developed. Infrared illumination is used to aid in segmentation of component lead and solder pad images from the background. Best fit lines through centroids of the lead and soldering pad images are calculated from which the displacement and angular orientation of the component relative to the pads is computed. Measurements are derived from a combination of slope and mean centroids from opposite sides of a device to reduce the effects of sensor resolution, missing pads, or missing leads. Since leads and pads are viewed simultaneously, the accuracy of the system is not dependent on precise positioning of the board within the inspection station. The system accuracy has been tested with 50 commercially manufactured printed circuit boards containing 500 samples of J-leaded plastic leaded chip carriers and small-outline integrated circuits of various sizes. The positions of the components were measured manually and are compared with measurements determined by the system.>

An experimental color vision inspection system for the detection of solder joint flaws on printed circuit boards is described. A tiered lighting arrangement is used to generate color contours on the solder joint for the detection and classification of defects including no solder, insufficient or excess solder, poor wetting of component leads or solder pad, and faults due to improper insertion of component leads. Each type of defect gives rise to a characteristic pattern of color contours which are processed using binary image techniques on each color plane of the image. Geometric descriptors measuring the shape of the color contours and the color level intensities of the solder joint images are used to identify each defect class. Statistical distributions are determined for each of the good and defective joint classes, from which a flaw detection and classification tree is derived. The performance of the system was tested on samples of commercially manufactured, wave-soldered circuit boards.>