In this paper, an application of simultaneous localization and mapping (SLAM) combined with visual reality (VR) is proposed to let the user experience and interact with an real environment through the headset with a mobile phone such that the user can easily visit the environment with cheaper equipment at any time. Compared with exploring an environment with general 2D map, the user can view the environment more realistically in an immersive manner. Here, an intensive map of environment with huge amount of point clouds, which is generated by employing the RTAB-Map SLAM with RGB-D cameras, is utilized to provide a more complete observation to the user than that of the sparse map. Since the memory of a mobile phone is limited and smaller than a personal computer, the data of whole map cannot be transmitted to the mobile phone once for VR displaying. Based on the architecture of memory management for detecting loop closure in the RTAB-Map SLAM, an efficient memory management for extracting the point cloud data to achieve the real-time VR displaying performance is designed by utilizing the Bayesian filtering and data clustering. The proposed approaches have been validated and analyzed in the experiments to present that our method can effectively reduce the memory usage for continuously and smoothly displaying the virtual scene.

In this paper, a robust visual tracking system with occlusion handling is proposed to track the target with real-time performance. The thermal camera, which can observe the heat originated from the target such as the human body or vehicle, can collaborate with the color camera to track the target in the cluttered environment or under occlusion. Unlike the general tracking by using the color camera and the thermal camera, which simply verifies the target hypotheses in these two kinds of image domains, a sampling multiple importance resampling scheme is proposed here to efficiently generate the hypotheses and verify them. The better hypotheses in the color and thermal images are selected to evaluate the sparse appearance representation such that the target under severe occlusion can be identified with real-time performance. Using this resampling scheme, the diversity and the convergency are simultaneously considered by adaptively fusing the hypotheses in the color and thermal images. Moreover, the updating strategy of target image model is designed by estimating the occlusion ratio and the environmental similarity such that the robustness of tracking can be greatly increased. Finally, the proposed approaches have been validated in several scenes to present the tracking performance.

This paper proposes a real-time indoor surveillance system which installs multiple depth cameras from vertical top-view to track humans. This system leads to a novel framework to solve the traditional challenge of surveillance through tracking of multiple persons, such as severe occlusion, similar appearance, illumination changes, and outline deformation. To cover the entire space of indoor surveillance scene, the image stitching based on the cameras' spatial relation is also utilized. The background subtraction of the stitched top-view image can then be performed to extract the foreground objects in the cluttered environment. The detection scheme including the graph-based segmentation, the head hemiellipsoid model, and the geodesic distance map are cascaded to detect humans. Moreover, the shape feature based on diffusion distance is designed to verify the human tracking hypotheses within particle filter. The experimental results demonstrate the real-time performance and robustness in comparison with several state-of-the-art detection and tracking algorithms.

This paper presents the visual servoing methodology that lands a micro aerial vehicle on a ground platform in the indoor environments. Since the space in the indoor environment is usually limited, both of the size of micro aerial vehicle and landing platform has to be small. Thus, the visual servoing based on two pin-hole cameras mounted on the front and belly of micro aerial vehicle is suitable for such scenario. The visual feature extraction of the landing platform is proposed to detect the specific image pattern on it. The flying control strategies with the camera switching are developed to search and approach the landing platform. The aligning control during the landing procedure is designed by the fuzzy logic control with the target position error on image. The overall performance of the landing algorithm has been validated in the online experiments.

In this paper, a vision based adaptive controller is proposed for the obstacle avoidance of a micro aerial vehicle (MAV) by using the optical flow information. In order to employ the optical flow for indicating the distance between the MAV and surrounding, the multi-thread processing algorithm is proposed to reliably obtain the optical flow information uniformly diffused in the whole image frame. The MAV system is treated as a switched system while manipulating the different modes during the obstacle avoidance task. Based on the desired flying direction of obstacle avoidance obtained by the optical flow estimation, we design an adaptive controller such that the desired trajectory can be tracked under different switching modes. The simulations present the tracking response of the adaptive controller of a switched system, and the experiments of the overall system validate the collision-avoidance performance of the MAV.

This research presents a 3-D human arms tracking method with a monocular camera. In our previous work, multiple clues have been integrated by the multiple importance sampling particle filter to track the arms with arbitrary motion on the images. Due to the lack of depth information when using a monocular camera, an online sequential pose estimation based on the structure-from-motion is proposed here to provide the 3D arm posture hypotheses of multiple importance sampling particle filter. In our 3D sequential pose estimating algorithm, the structure of each arm parts is priory assumed to be a volumetric model with the points uniformly distributed on its surface. The 3D motion of arms can be recovered more reliably by visual tracking of the points on the known structure of each arm. In addition, while the objects in view appear larger when they are closer to the camera, the size effect from perspective projection is considered to disambiguate the arm posture. The posture hypotheses from the multiple importance sampling are finally verified by 2D visual features and the 3D posture information augmented with the size effect. The robustness and efficiency of the proposed algorithm have been validated in the experiments.

This paper presents a head and hands tracking method with a monocular camera for human machine interaction (HMI). The targets are tracked independently when they are far from each other, however, they are merged with dependent likelihood measurements in higher dimension while they are likely to interrupt each other. When tracking one target in the independent situation, other targets are masked to decrease the disturbances of skin color on the tracked one. Multiple clues, including the combination of the locally discriminative color weighted image and the back-projection image of the reference color model, the motion history image and the gradient orientation feature, are employed to verify the hypotheses originated from the particle filter. On the other hand, when the head and hands are closing or even overlapping, the multiple importance sampling (MIS) particle filter generates the tracking hypotheses of merged targets by the skin blob mask and the depth order estimation. These merged hypotheses are then evaluated by the visual cues of occluded face template, hand shape orientation and motion continuity. The experimental results present the real-time efficiency and the robustness in comparison with the OpenNI tracker which has been released recently for the Kinect sensor.

This paper presents a tracking-based surveillance system that is capable of tracking multiple moving objects, with almost real-time response, through the effective cooperation of multiple pan-tilt cameras. To construct this surveillance system, the distributed camera agent, which tracks multiple moving objects independently, is first developed. The particle filter is extended with target depth estimate to track multiple targets that may overlap with one another. A strategy to select the suboptimal camera action is then proposed for a camera mounted on a pan-tilt platform that has been assigned to track multiple targets within its limited field of view simultaneously. This strategy is based on the mutual information and the Monte Carlo method to maintain coverage of the tracked targets. Finally, for a surveillance system with a small number of active cameras to effectively monitor a wide space, this system is aimed to maximize the number of targets to be tracked. We further propose a hierarchical camera selection and task assignment strategy, known as the online position strategy, to integrate all of the distributed camera agents. The overall performance of the multicamera surveillance system has been verified with computer simulations and extensive experiments.

This paper presents an upper body tracking algorithm with a single monocular camera. In order to be suitable for human robot interaction, the designed method should be free to work on the moving camera platform and also can achieve real-time performance. The dimension of human posture model is extremely high, and we hereby focus on the visual extraction of head and arms. A hierarchical structure model is proposed to solve the tracking problem by particle filter with partitioned sampling in the order of head, upper arm and the forearm. The hand position, straight edge of arm and temporal information are combined by the multiple importance sampling particle filter to efficiently estimate the irregular gesture of arms on image frames. The visual clues of the motion, appearance and shape to human face and arms are to verify the various hypotheses from the multiple importance sampling schemes. To validate the effectiveness of the proposed tracking approach, extensive experiments have been performed, of which the results appear to be quite promising.

Tracking the pose of human face has long been an important research topic which has many important applications, and it is particularly challenging with a monocular camera because the depth information is lost due to the perspective projection. This work adopts particle filter with partitioned sampling to decompose the state space of face pose tracking into two subspaces for increasing the sampling efficiency, thus achieving satisfactory performance with fewer particles. The parameters in the first subspace describe the target on image plane, and the parameter in the second subspace is used for the estimate of the face pose in yaw angle direction. For the evaluation of each hypothesis in the second subspace, a statistical learning algorithm called relevance vector machine (RVM) is used to map a face containing image to the pose of the face. The training of RVM is tailored to each detected frontal face, and it takes less than half second, which is suitable for a real-time application. The learning based regression model also presents the insensitive ability to expression variation and unmodeled degree of freedom. The experimental results verify that the combination of particle filter and RVM can efficiently reduce the processing time and add robustness to the performance of the system, thus making this algorithm applicable to human-machine interface with low-cost webcams.

This research presents an upper body tracking method with a monocular camera. The human model is defined in a high dimensional state space. We hereby propose a hierarchical structure model to solve the tracking problem by SIR (Sampling Importance Resampling) particle filter with partitioned sampling. The image spatial and temporal information is used to track the human body and estimate the human posture. When doing the human-machine interaction, a static monocular camera may not get plenty of information from 2D images, so we must move the camera platform to a better position for acquiring more enriched image information. The proposed upper body tracking technique will then adjust to estimating the human posture during the camera moving. To validate the effectiveness of the proposed tracking approach, extensive experiments have been performed, of which the result appear to be quite promising.

This paper presents a nearly real-time surveillance system to track multiple moving objects by controlling multiple pan-tilt camera platforms. In order to describe the relationship between the targets and camera in this surveillance system, the input/output hidden Markov model (HMM) is applied here in the well-defined spherical camera coordinate. For the less number of cameras to effectively monitor a wide surveillance space, the overall cameras have to closely cooperate. We propose a hierarchical camera selection and task distribution strategy, and the action decision of each camera platform is according to its assigned role. Furthermore, an optimal camera action selection strategy is presented for one camera which is assigned to track multi-target within its limited field of view. The maximization of mutual information for the action design is evaluated by the Monte Carlo method. The overall performance has been validated in the experiments of real-time surveillance.

This paper presents a real-time surveillance system to track multiple moving objects by controlling a pan-tilt camera platform. In order to describe the relationship between the targets and camera in this surveillance system, the input/output hidden Markov model (HMM) is applied here in the well-defined spherical camera coordinate. Since the targets are hard to be distinguished with one single camera when they are close to each other, we extend the particle filter for multitarget tracking with depth level estimate to track interacting targets. The targets overlapping each other still can be tracked in the images captured by single camera. Furthermore, an optimal camera action selection strategy is proposed to track multitarget within its limited field of view. The maximization of mutual information for the action design is formalized and implemented by the Monte Carlo method. The overall performance has been validated in the experiments of real-time tracking.

This paper proposes a robust visual tracking framework to track circle-like objects in cluttered environment. Instead of using the resulting positions after circle detection and tracking in the image domain, we directly perform the visual tracking task in the parameter space which describes the measurement features. The visual tracking technique is combined with targets generation closely. We utilize the probabilistic data association filter (PDAF) to filter the detected measurements with noise and disturbance. The likelihood ratio through the Hough transform is employed to modify the evaluation of the association probability and make the estimate more reliable. Furthermore, the joint probabilistic data association filter (JPDAF) is used to deal with the multiple circle-like objects tracking. The likelihood variation of each target is introduced with JPDAF as a basis of the predictions for different targets. The overall performance has been verified in several challenging experiments

This paper aims to track multiple people in a multicamera system for cluttered environment which can be divided into two important parts: one is tracking multiple people in a single camera environment, and the other is tracking multiple people in a multi-camera environment. In a single camera environment, we apply the motion detector and the ellipse algorithm to detect a new person intruding the surveillance area. Then, we utilize the template matching and the ellipse matching to track the person. To prevent tracking failure when people cross over each other, we include the hereby proposed joint visual probabilistic data association filter (JVPDAF) to track multiple people successfully. In a multi-camera environment, the major problem is to determine whether the new person intruding into some surveillance area of a camera is an identified person by some other camera or not. To resolve the aforementioned problem, we propose an approach called consistence labeling. After such labeling process, we track this person by the JVPDA algorithm. Finally, effectiveness of this tracking algorithm is validated via extensive experiments

The paper proposes a new fuzzy data-mining algorithm for extracting interesting knowledge from quantitative transactions stored as object data. Each item itself is thought of as a class, and each item purchased in a transaction is thought of as an instance. Instances with the same class (item name) may have different quantitative attribute values since they may appear in different transactions. The proposed fuzzy algorithm can be divided into two main phases. The first phase is called the fuzzy intra-object mining phase, in which the linguistic large itemsets associated with the same classes (items) but with different attributes are derived. The second phase is called the fuzzy inter-object mining phase, in which the large itemsets are derived and used to represent the relationship among different kinds of objects. Experimental results also show the effects of the proposed algorithm.

In visual tracking, Multi-target tracking (MTT) systems encounter the problem that unavoidably moving targets may overlap each other and the measurement process of each target becomes dependent, so we construct a tracking system with considering joint image likelihood to track recognize targets, even homogeneous ones. Also, in order to enhance the tracking performance, we extend the sequential importance sampling (SIS) particle filter with the separated importance functions for tracking each target and detection at the same time. The overall performance is validated in the experiments and shows the robustness with near real-time tracking.

Visual tracking in cluttered environments is attractive and challenging. This paper establishes a probabilistic framework, called the visual probabilistic data-association filter (VPDAF), to deal with this problem. The algorithm is based on the probabilistic data-association method for estimating a true target from a cluster of measurements. There are two other key concepts which are involved in VPDAF. First, the sensor data are visual, similar to the target in the image space, which is a crucial property that should not be ignored in target estimation. Second, the traditional probabilistic data-association filter for the underlying application is vulnerable to stationary disturbances in image space, mainly due to some annoying background scenes which are rather similar to the target. Intuitively, such persistent noises should be separated out and cleared away from the continuous measurement data for seeking successful target detection. The proposed VPDAF framework, which incorporates template matching, can achieve the goal of reliable realtime visual tracking. To demonstrate the superiority of the system performance, extensive yet challenging experiments have been conducted.

This paper presents the development of a visual tracking and visual servo system for a reconnoitering air vehicle. The air vehicle is mounted with a camera platform underneath. It is proposed to perform an aerial reconnoitering mission for continuously viewing an interesting target. The vision system is composed of a hybrid visual tracking system and an image based fuzzy logic visual servo system. The designed hybrid visual tracking system can track arbitrary-shaped moving object. Also, the followed visual servo system is to control the air vehicle hovering the target autonomously. The overall system is simulated and experimented in a virtual reality environment to validate the research work.