Conventional VR experiences rely heavily on visual input, creating barriers for individuals who are blind or have low vision (BLV). We present a mobile VR system that enables BLV users to explore virtual environments independently using a smartphone as a simulated white cane, augmented with spatial audio and haptic feedback. Built on ubiquitous consumer devices, the system supports embodied exploration and encourages physical movement, with multimodal feedback to help users transfer real-world cane techniques into virtual spaces and build mental maps. A user study shows that BLV participants can independently navigate complex virtual environments using familiar cane strategies. We conclude with design recommendations for accessible mobile VR systems informed by our findings.

Ensuring safe and independent navigation poses considerable difficulties for individuals who are blind or have low vision (BLV), as it requires detailed knowledge of their immediate environment. Our research highlights the critical need for accessible data on sidewalk materials and objects, which is currently lacking in existing map services. To bridge this gap, we present the Sidewalk Material Data Acquisition Framework (SMDAF), designed for large-scale data collection. This framework includes (1) a lightweight data collection system embedded in a white cane, which captures audio data through the interaction of the cane tip with the sidewalk surface, and a mobile app that facilitates data storage and management, resulting in a novel multimodal dataset comprising both image and audio data; and (2) a unique Cross-Modal Knowledge Distillation (CMKD) technique for an enhanced audio material classifier. Our CMKD approach employs an image-based model as the teacher to improve the audio model, incorporating an Enhanced Bidirectional learning method with an intuitive filtering technique: Bidirectional Correct Sample Filtering (BCSF). BCSF filters correct samples to prevent the distillation of incorrect knowledge, addressing the issue of inaccurate cross-modal learning. This novel approach has resulted in a 1.84% improvement in Macro Accuracy, achieving an overall accuracy of 87.62%, surpassing all state-of-the-art KD and CMKD methods. This study underscores the efficacy of SMDAF and provides a practical CMKD technique for future cross-modal learning tasks. Code and dataset are available (https://github.com/FgSurewin/SMDAF-CMKD).

The corona virus pandemic has introduced limitations which were previously not a cause for concern. Chief among them are wearing face masks in public and constraints on the physical distance between people as an effective measure to reduce the virus spread. Visual surveillance systems, which are common in urban environments and initially commissioned for security surveillance, can be re-purposed to help limit the spread of COVID-19 and prevent future pandemics. In this work, we propose a novel integration technique for real-time pose estimation and multiple human tracking in a pedestrian setting, primarily for social distancing, using CCTV camera footage. Our technique promises a sizeable increase in processing speed and improved detection in very low-resolution scenarios. Using existing surveillance systems, pedestrian pose estimation, tracking and localization for social distancing (PETL4SD) is proposed for measuring social distancing, which combines the output of multiple neural networks aided with fundamental 2D/3D vision techniques. We leverage state-of-the-art object and pose estimation algorithms, combining their strengths, for increase in speed and improvement in detections. These detections are then tracked using a bespoke version of the FASTMOT algorithm. Temporal and analogous estimation techniques are used to deal with occlusions when estimating posture. Projective geometry along with the aforementioned posture tracking is then used to localize the pedestrians. Inter-personal distances are calculated and locally inspected to detect possible violations of the social distancing rules. Furthermore, a "smart violations detector" is employed which estimates if people are together based on their current actions and eliminates false social distancing violations within groups. Finally, distances are intuitively visualized with the right perspective. All implementation is in real time and is performed on Python. Experimental results are provided to validate our proposed method quantitatively and qualitatively on public domain datasets using only a single CCTV camera feed as input. Our results show our technique to outperform the baseline in speed and accuracy in low-resolution scenarios. The code of this work will be made publicly available on GitHub at https://github.com/bilalze/PETL4SD.

In this work, a storefront accessibility image dataset is collected from Google street view and is labeled with three main objects for storefront accessibility: doors (for store entrances), doorknobs (for accessing the entrances) and stairs (for leading to the entrances). Then MultiCLU, a new multi-stage context learning and utilization approach, is proposed with the following four stages: Context in Labeling (CIL), Context in Training (CIT), Context in Detection (CID) and Context in Evaluation (CIE). The CIL stage automatically extends the label for each knob to include more local contextual information. In the CIT stage, a deep learning method is used to project the visual information extracted by a Faster R-CNN based object detector to semantic space generated by a Graph Convolutional Network. The CID stage uses the spatial relation reasoning between categories to refine the confidence score. Finally in the CIE stage, a new loose evaluation metric for storefront accessibility, especially for knob category, is proposed to efficiently help BLV users to find estimated knob locations. Our experiment results show that the proposed MultiCLU framework can achieve significantly better performance than the baseline detector using Faster R-CNN, with +13.4% on mAP and +15.8% on recall, respectively. Our new evaluation metric also introduces a new way to evaluate storefront accessibility objects, which could benefit BLV group in real life.

Gatherings of thousands to millions of people frequently occur for an enormous variety of educational, social, sporting, and political events, and automated counting of these high-density crowds is useful for safety, management, and measuring significance of an event. In this work, we show that the regularly accepted labeling scheme of crowd density maps for training deep neural networks may not be the most effective one. We propose an alternative inverse k-nearest neighbor (i[Formula: see text]NN) map mechanism that, even when used directly in existing state-of-the-art network structures, shows superior performance. We also provide new network architecture mechanisms that we demonstrate in our own MUD-i[Formula: see text]NN network architecture, which uses multi-scale drop-in replacement upsampling via transposed convolutions to take full advantage of the provided i[Formula: see text]NN labeling. This upsampling combined with the i[Formula: see text]NN maps further improves crowd counting accuracy. We further analyze several variations of the i[Formula: see text]NN labeling mechanism, which apply transformations on the [Formula: see text]NN measure before generating the map, in order to consider the impact of camera perspective views, image resolutions, and the changing rates of the mapping functions. To alleviate the effects of crowd density changes in each image, we also introduce an attenuation mechanism in the i[Formula: see text]NN mapping. Experimentally, we show that inverse square root [Formula: see text]NN map variation (iR[Formula: see text]NN) provides the best performance. Discussions are provided on computational complexity, label resolutions, the gains in mapping and upsampling, and details of critical cases such as various crowd counts, uneven crowd densities, and crowd occlusions.

To alleviate the cost of collecting and annotating large-scale "3D object" point cloud data, we propose an unsupervised learning approach to learn features from an unlabeled point cloud dataset by using part contrasting and object clustering with deep graph convolutional neural networks (GCNNs). In the contrast learning step, all the samples in the 3D object dataset are cut into two parts and put into a "part" dataset. Then a contrast learning GCNN (ContrastNet) is trained to verify whether two randomly sampled parts from the part dataset belong to the same object. In the cluster learning step, the trained ContrastNet is applied to all the samples in the original 3D object dataset to extract features, which are used to group the samples into clusters. Then another GCNN for clustering learning (ClusterNet) is trained from the orignal 3D data to predict the cluster IDs of all the training samples. The contrasting learning forces the ContrastNet to learn semantic features of objects, while the ClusterNet improves the quality of learned features by being trained to discover objects that belong to the same semantic categories by using cluster IDs. We have conducted extensive experiments to evaluate the proposed framework on point cloud classification tasks. The proposed unsupervised learning approach obtains comparable performance to the state-of-the-art with heavier shape auto-encoding unsupervised feature extraction methods. We have also tested the networks on object recognition using partial 3D data, by simulating occlusions and perspective views, and obtained practically useful results. The code of this work is publicly available at: https://github.com/lingzhang1/ContrastNet.

This paper describes a study on automated emotion recognition using four different modalities - audio, video, electromyography (EMG), and electroencephalography (EEG). We collected a dataset using the 4 modalities as 12 human subjects expressed six different emotions or maintained a neutral expression. Three different aspects of emotion recognition were investigated: model selection, feature selection, and data selection. Both generative models (DBNs) and discriminative models (LSTMs) were applied to the four modalities, and from these analyses we conclude that LSTM is better for audio and video together with their corresponding sophisticated feature extractors (MFCC and CNN), whereas DBN is better for both EMG and EEG. By examining these signals at different stages (pre-speech, during-speech, and post-speech) of the current and following trials, we found that the most effective stages for emotion recognition from EEG occur after the emotion has been expressed, suggesting that the neural signals conveying an emotion are long-lasting.

In this work, we use a generative adversarial network (GAN) to train crowd counting networks using minimal data. We describe how GAN objectives can be modified to allow for the use of unlabeled data to benefit inference training in semi-supervised learning. More generally, we explain how these same methods can be used in more generic multiple regression target semi-supervised learning, with crowd counting being a demonstrative example. Given a convolutional neural network (CNN) with capabilities equivalent to the discriminator in the GAN, we provide experimental results which show that our GAN is able to outperform the CNN even when the CNN has access to significantly more labeled data. This presents the potential of training such networks to high accuracy with little data. Our primary goal is not to outpreform the state-of-the-art using an improved method on the entire dataset, but instead we work to show that through semi-supervised learning we can reduce the data required to train an inference network to a given accuracy. To this end, systematic experiments are performed with various numbers of images and cameras to show under which situations the semi-supervised GANs can improve results.

In this paper, we propose a deep learning based approach for facial action unit (AU) detection by enhancing and cropping regions of interest of face images. The approach is implemented by adding two novel nets (a.k.a. layers): the enhancing layers and the cropping layers, to a pretrained convolutional neural network (CNN) model. For the enhancing layers (noted as E-Net), we have designed an attention map based on facial landmark features and apply it to a pretrained neural network to conduct enhanced learning. For the cropping layers (noted as C-Net ), we crop facial regions around the detected landmarks and design individual convolutional layers to learn deeper features for each facial region. We then combine the E-Net and the C-Net to construct a so-called Enhancing and Cropping Net (EAC-Net), which can learn both features enhancing and region cropping functions effectively. The EAC-Net integrates three important elements, i.e., learning transfer, attention coding, and regions of interest processing, making our AU detection approach more efficient and more robust to facial position and orientation changes. Our approach shows a significant performance improvement over the state-of-the-art methods when tested on the BP4D and DISFA AU datasets. The EAC-Net with a slight modification also shows its potentials in estimating accurate AU intensities. We have also studied the performance of the proposed EAC-Net under two very challenging conditions: (1) faces with partial occlusion and (2) faces with large head pose variations. Experimental results show that (1) the EAC-Net learns facial AUs correlation effectively and predicts AUs reliably even with only half of a face being visible, especially for the lower half; (2) Our EAC-Net model also works well under very large head poses, which outperforms significantly a compared baseline approach. It further shows that the EAC-Net works much better without a face frontalization than with face frontalization through image warping as pre-processing, in terms of computational efficiency and AU detection accuracy.

Action Unit (AU) detection becomes essential for facial analysis. Many proposed approaches face challenging problems in dealing with the alignments of different face regions, in the effective fusion of temporal information, and in training a model for multiple AU labels. To better address these problems, we propose a deep learning framework for AU detection with region of interest (ROI) adaptation, integrated multi-label learning, and optimal LSTM-based temporal fusing. First, an ROI cropping net is designed to make sure specific interested regions of faces are learned independently, each sub-region has a local convolutional neural network (CNN) whose convolutional filters will only be trained for the corresponding region. Second, multi-label learning is employed to integrate the outputs of those individual ROI cropping nets, which learns the inter-relationships of various AUs and acquires global features across sub-regions for AU detection. Finally, the optimal selection of multiple LSTM layers are carried out to best fuse temporal features, in order to make the AU prediction the most accurate. The proposed approach is evaluated on two popular AU detection datasets, BP4D and DISFA, outperforming the state of the art significantly, with an average improvement of around 13% in BP4D and 25% in DISFA, respectively.

In this paper, we propose a deep learning based approach for facial action unit detection by enhancing and cropping the regions of interest. The approach is implemented by adding two novel nets (layers): the enhancing layers and the cropping layers, to a pretrained CNN model. For the enhancing layers (the E-Net), we designed an attention map based on facial landmark features and applied it to a pretrained neural network to conduct enhanced learning. For the cropping layers (the C-Net), we crop facial regions around the detected landmarks and design convolutional layers to learn deeper features for each facial region. We then fuse the E-Net and the C-Net to obtain our Enhancing and Cropping (EAC) Net, which can learn both feature enhancing and region cropping functions. Our approach shows significant improvement in performance compared to the state-of-the-art methods applied to BP4D and DISFA AU datasets.

In this paper, we describe our proposed approach for participating in the Third Emotion Recognition in the Wild Challenge (EmotiW 2015). We focus on the sub-challenge of Audio-Video Based Emotion Recognition using the AFEW dataset. The AFEW dataset consists of 7 emotion groups corresponding to the 7 basic emotions. Each group includes multiple videos from movie clips with people acting a certain emotion. In our approach, we extract LBP-TOP-based video features, openEAR energy/spectral-based audio features, and CNN (convolutional neural network) based deep image features by fine-tuning a pre-trained model with extra emotion images from the web. For each type of features, we run a SVM grid search to find the best RBF kernel. Then multi-kernel learning is employed to combine the RBF kernels to accomplish the feature fusion and generate a fused RBF kernel. Running multi-class SVM classification, we achieve a 45.23% test accuracy on the AFEW dataset. We then apply a decision optimization method to adjust the label distribution closer to the ground truth, by setting offsets for some of the classifiers' prediction confidence score. By applying this modification, the test accuracy increases to 50.46%, which is a significant improvement comparing to the baseline accuracy 39.33% .

Capturing aerial imagery at high resolutions often leads to very low frame rate video streams, well under full motion video standards, due to bandwidth, storage, and cost constraints. Low frame rates make registration difficult when an aircraft is moving at high speeds or when global positioning system (GPS) contains large errors or it fails. We present a method that takes advantage of persistent cyclic video data collections to perform an online registration with drift correction. We split the persistent aerial imagery collection into individual cycles of the scene, identify and correct the registration errors on the first cycle in a batch operation, and then use the corrected base cycle as a reference pass to register and correct subsequent passes online. A set of multi-view panoramic mosaics is then constructed for each aerial pass for representation, presentation and exploitation of the 3D dynamic scene. These sets of mosaics are all in alignment to the reference cycle allowing their direct use in change detection, tracking, and 3D reconstruction/visualization algorithms. Stereo viewing with adaptive baselines and varying view angles is realized by choosing a pair of mosaics from a set of multi-view mosaics. Further, the mosaics for the second pass and later can be generated and visualized online as their is no further batch error correction.

Moving vehicle detection and classification using multimodal data is a challenging task in data collection, audio-visual alignment, and feature selection, and effective vehicle classification in uncontrolled environments. In this work, we first present a systematic way to align the multimodal data based the multimodal temporal panorama generation. Then various types of features are extracted to represent diverse and multimodal information. Those include global geometric features (aspect ratios, profiles), local structure features (HOGs), various audio features in both spectral and perceptual representations. A flexible sequential forward selection algorithm with multi-branch searching is used to select a set of important features at different levels of feature combinations. Finally, using the same datasets for two different classification tasks, we show that the roles of audio and visual features are task-specific. Furthermore, in both cases, the combination of some of the features with multimodal and complementary information can improve the accuracy than using the individual features only. Therefore finer and more accurate classification can be achieved by two different levels of integration: feature level and the decision level.

Vehicle images captured by traffic and surveillance video cameras in various conditions usually exhibit several unexpected variations that worsen vehicle classification. These factors include occlusions, motion blur, and changes in perspective views. Complete and normalized views of vehicle images, if being able to reconstructed from the unsatisfactory data, will facilitate more accurate data labeling, feature extraction and multi-class vehicle classification. We propose a multimodal temporal panorama (MTP) approach to accurately extracting and reconstructing moving vehicles in real-time using a remote multimodal (audio/video) monitoring system. The MTP representation consists of: 1) a panoramic view image (PVI) for detecting vehicles using the concept of 1D vertical detection line; 2) an epipolar plane image (EPI), generated from 1D epipolar lines along the vehicles' moving paths, to characterize their speeds and directions; and 3) corresponding audio signals collected at the vehicle detection point to reduce false target detection in the PVI. Using the MTP approach, reconstructed vehicles all have the same side views, with less or no occlusions and motion blur. Using SVM classifiers for multiclass problems indicates that the classification accuracy using reconstruction results improves about 10% over that using corresponding vehicle images from original video for a dataset of about 140 vehicles. Our ultimate goal is to use the audio-visual vehicle data for multimodal vehicle classification and anomaly detection.

We propose a content-based 3-D mosaic (CB3M) representation for long video sequences of 3-D and dynamic urban scenes captured by a camera on a mobile platform. In the first phase, a set of parallel-perspective (pushbroom) mosaics with varying viewing directions is generated to capture both the 3-D and dynamic aspects of the scene under the camera coverage. In the second phase, a segmentation-based stereo matching algorithm is applied to extract parametric representations of the color, structure and motion of the dynamic and/or 3-D objects in urban scenes, where a lot of planar surfaces exist. Multiple pairs of stereo mosaics are used for facilitating reliable stereo matching, occlusion handling, accurate 3-D reconstruction, and robust moving target detection. CB3M is a highly compressed visual representation for a dynamic 3-D scene, and has object contents of both 3-D and motion information. Experimental results are given for various real video sequences of large-scale 3-D scenes.

In this paper we propose a fast method for constructing multi-view stereo panoramas using a layering approach. Constructing panoramas requires accurate camera pose estimation and will often require an image blending or interpolation method to generate seamless results. We use a registration error correction method that provides globally corrected and fast results for paths that create cycles such as circular paths, back and forth straight sweeps, and even a single sweep. Then we apply our layering approach to generate multi-view stereo panoramas quickly for time sensitive applications that require immediate results and 3D perception.

In this work, we present a multimodal temporal panorama (MTP) representation that synchronizes visual, motion, and acoustic signatures of moving vehicles in the time axis. The MTP representation includes two layers: a synopsis layer and a snapshot layer. The temporal synopsis consists of 1) a panoramic view image (PVI) to represent vehicles' presence, which is constructed from 1D vertical detecting lines of a selected column location of all video frames, 2) an epipolar plane image (EPI) to characterize their motion (speeds and directions), generated from 1D horizontal scanning lines along the vehicles' moving paths, and 3) an audio wave scroll for visualizing moving vehicles' acoustic signatures. The MTP synopsis not only synchronizes all the three modalities (visual, motion and acoustic) of the vehicles, but also provides information that can perform automatic detection tasks including moving vehicle visual detection, motion estimation, and acoustic signature retrieval. Then in the snapshot layer, the occlusion-free, motion-blur-free, and view-invariant reconstruction of each vehicle (with both shape and motion information) and its acoustic signatures (e.g. spectrogram) are embedded. The MTP provides a very effective approach to (semi-)automatically labeling the multimodal data of uncontrolled traffic scenes in real time for further vehicle classification, check-point inspection and traffic analysis. The concept of MTP may not be only limited to visual, motion and audio modalities, it could also be applicable to other sensing modalities that can obtain data in the temporal domain.

Laser Doppler Vibrometers (LDVs) have been widely applied for detecting vibrations in applications such as mechanics, bridge inspection, biometrics, as well as long-range surveillance in which acoustic signatures can be obtained at a large distance. However, in both industrial and scientific applications, the LDVs are manually controlled in surface selection, laser focusing, and acoustic acquisition. In this paper, we propose an active stereo vision approach to facilitate fast and automated laser pointing and tracking for long-range LDV hearing. The system contains: 1) a mirror on a Pan-Tilt-Unit (PTU) to reflect the laser beam to any locations freely and quickly, and 2) two Pan-Tilt-Zoom (PTZ) cameras, one of which is mounted on the Pan-Tilt-Unit (PTU) and aligned with the laser beam synchronously. The distance measurement using the stereo vision system as well as triangulation between camera and the LDV laser beam allow us to fast focus the laser beam on selected surfaces and to obtain acoustic signals up to 200 meters in real time. We present some promising results with the collaborative visual and LDV measurements for laser pointing and focusing in order to achieve long range audio detection.

This paper presents a novel technique to register a set of two 3D laser scans obtained from a ground robot and a wall-climbing robot which operates on the ceiling to construct a complete map of the indoor environment. Traditional laser scan registration methods like the Iterative Closest Point (ICP) algorithm will not converge to a global minimum without a good initial estimate of the transformation matrix. Our technique uses an overhead camera on the wall-climbing robot to keep line of sight with the ground robot and solves the Perspective Three Point (P3P) Problem to obtain the transformation matrix between the wall-climbing robot and the ground robot, which serves as a good initial estimate for the ICP algorithm to further refine the transformation matrix. We propose a novel particle filter algorithm to identify the real pose of the wall-climbing robot out of up to four possible solutions to P3P problem using Grunert's algorithm. The initial estimate ensures convergence of the ICP algorithm to a global minimum at all times. The simulation and experimental results indicate that the resulting composite laser map is accurate. In addition, the vision-based approach increases the efficiency by reducing the number of iterations of the ICP algorithm.

Finding traversable paths using computer vision is one of the most important components of an intelligent mobile robot system. For a wall climbing robot that operates in an urban environment, it is essential to automatically detect surface types and orientations for switching between moving and climbing, and for applying different adhesive forces both to save energy and ensure its own safety. This paper presents a novel segmentation-based stereovision approach in order to rapidly obtain accurate 3D estimations of urban scenes with largely textureless areas and sharp depth changes. The new approach takes advantage of the fact that many man-made objects in an urban setting consist of planar surfaces. Our approach has three main components: extraction of natural (planar) matching primitives, stereo matching via three-step algorithm (global match, local match and plane fitting), and plane merging and parameter refinement. Experimental results are provided for real indoor scenes.

This paper presents a novel method to construct a complete 3D map that includes all surfaces (ceiling, wall, and furniture tops, etc.) in indoor environments. A team of four robots, including three ground robots and one wall-climbing robot is deployed in a tetrahedron configuration that satisfies the perspective three point (P3P) problem. P3P problem is to estimate the pose of a perspective camera on the wall-climbing robot viewing three ground robots, which will produce up to four solutions using Grunert's algorithm while only one of them is genuine. We propose a probabilistic Bayesian algorithm that identifies the unique solution of the P3P problem using the mobility of the camera. Based on this technique, we introduce an intra-robot localization method to determine the geometric relationship among four robots. Each ground robot is equipped with a rotary laser range finder (LRF), a pan-tilt-zoom camera, and a LED cluster. The wall-climbing robot is fitted with a LRF, a perspective camera, and a motion sensor. Through the vision sensors, the robots obtain their relative poses by solving the P3P problem. Through the LRF on each robot, 4 laser point cloud maps are produced from each robot's point of view. With the information of relative poses of the multiple robots and the calibration data of each LRF and camera pair, the 4 partial maps are fused to acquire a complete 3D map that is rich with information of all surfaces. Our approach outperforms the traditional range image fusion algorithms in terms of time complexity and is suitable for real-time implementation. Real experiments verified the effectiveness of the method.

Currently available methods for object recognition and classification primarily rely on static information in single-frame images. However, for the combat aerial video (usually low resolution video), all these static indexes used for object classification and recognition are almost impossible to obtain. To address this challenge, we propose an innovative 3D and dynamic semantic scene analysis based approach that exploits surveillance video data mainly captured from UAV platforms to classify static object (e.g. buildings) and moving object (e.g. vehicles) automatically. In our proposed automatic object detection and classification framework, in addition to 3D static object's visual features (e.g. building's or vehicle's shape, line orientation, color, and texture) and the 3D static structures of the urban environment, we also explore dynamic video features which include vehicle motion patterns over time. All these static and dynamic features will be considered to construct spatial-temporal feature vectors, and the new generated vectors will then be sent to a probabilistic dynamic influence diagram (DID) reasoning model for real-time and automatic building and vehicle classification. In addition, we also propose novel 3D algorithms on automatic building detection, 3D terrain modeling, and visualization to support accurate object categorization/classification.

Sentient computing uses networks of sensors to capture and maintain an internal representation ("world model") of an indoor environment, thereby allowing applications to have greater awareness of users and their requirements. This chapter shows how computer vision information obtained by several cameras can be used to enhance the capabilities of a sentient computing system which previously relied on ultrasound to track people and devices. Integration is achieved at the system level through the metaphor of shared perceptions in the sense that the different modalities are guided by and provide updates for a shared internal model. This world model incorporates aspects of both the static (e.g. positions of office walls and doors) and dynamic (e.g. location and appearance of devices and people) environments. It serves both as an ontology of prior information and as a source of context which is shared between applications. Fusion and inference are performed by Bayesian networks which model the probabilistic dependencies and reliabilities of different sources of information over time. It is shown that the fusion process significantly enhances the capabilities and robustness of the system, thus enabling it to maintain a richer and more accurate world model.

Multimodal sensing, data processing, and system integration are still in a fast growing stage of research and development. Much work needs to be done to ensure successful real-world, large-scale surveillance applications. The collection in this book is just samples of some representative works in the field and therefore by no means to be complete. However, we hope this collection will stimulate more interests in the research and development of multimodal surveillance techniques, including the further developments of multimodal sensors, multimodal data fusion algorithms, and multimodal surveillance systems.

Recent improvements in laser vibrometry and day/night infrared (IR) and electro-optical (EO) imaging technology have created the opportunity to create a long-range multimodal surveillance system. This multimodal capability would greatly improve security force performance through clandestine listening of targets that are probing or penetrating a perimeter defense. This system could also provide the feeds for advanced face and voice recognition systems.

This paper presents a new approach to the intralocalization among a team of robots working in constrained 3D space of urban environments. As the base formation, a team of three ground robots and one wall-climbing robot are deployed on ground and on a wall or ceiling, respectively. The three ground robots localize themselves using an existing panoramic vision-based method. However, no existing method can uniquely determine the pose of the climbing robot based on the positions of three ground robots in its image and in the world coordinate systems; up to four valid solutions could exist using known algorithms, although only one is genuine. By carefully examining these methods, two new algorithms for uniquely locating the climbing robot are proposed. The first algorithm makes use of the straight line constraint of robot motion and can uniquely determine the pose of climbing robot by moving the climbing robot straightly for two small steps. The second algorithm is based on the principle of Bayesian filter and take advantage of the motion sensor readings to loose the straight line constraint. The algorithm could continuously determine the climbing robot’s pose after the initial pose is obtained. Extensive simulations are conducted to validate the soundness and robustness of our algorithms. Preliminary experiments are also carried out to examine the feasibility in applying these algorithms in real robot applications.

In this paper, stereo matching and 3D visualization issues are studied for a linear pushbroom stereo model built for 3D gamma-ray (or X-ray) cargo inspection. A fast and automated stereo matching algorithm based on free-form deformable registration is developed to obtain 3D measurements of the objects under inspection. For facilitate stereo matching, an adaptive window min-max method is used to enhance the contrasts and boundaries of the challenging gamma-ray images. For fast implementation, a multi-resolution approach is also applied. A user interface is designed for interactive 3D measurements and visualization of the objects in interests. Experimental results are presented for real gamma-ray images of a 3D cargo container and the objects inside. The 3D measurements could add more value to today's cargo inspection techniques

We have developed several basic components for our virtualized classroom project: automated data collection, intelligent media integration, and flexible user interfaces. As an example of media integration, we show that when the PPT slides are projected onto the digital whiteboard where the instructor annotates, modifies, or expands the PPT presentation, geometrically registering high-resolution images from the two sources yields high quality digital presentations (e-presentations). We use a low-cost digital camera as a bridge to align the two sources and present a hybrid registration approach to align the PPT and handwriting images. Experimental results are presented to validate our approach

Multimodal surveillance systems using visible/IR cameras and other sensors are widely deployed today for security purpose, particularly when subjects are at a large distance. However, audio information as an important data source has not been well explored. One of the reasons is because audio detection using microphones needs installation close to the subjects in monitoring. In this paper, we investigate a novel "optical" sensor, called laser Doppler vibrometer (LDV), for capturing voice signals in a very large range to realize a truly remote and multimodal surveillance system. Speech enhancement approaches are studied based on the characteristics of LDV audio. Experimental results show that remote voice detection via an LDV is promising when choosing appropriate targets close to human subjects in the environment

This paper presents a self-localization strategy for a team of heterogeneous mobile robots including ground mobile robots of various sizes and wall-climbing robots. These robots are equipped with various visual sensors, such as miniature webcams, omnidirectional cameras, and PTZ cameras. As the core of this work, a formation of four-robot team is constructed to operate in a 3D space, e.g., moving on ground, climbing on walls and clinging to ceilings. The four robots could dynamically localize themselves asynchronously by employing cooperative vision techniques. Three of them on the ground mutually view each other and determine their relative poses with 6 degrees of freedom (DOFs). A wall-climbing robot, which significantly extends the work space of the robot team to 3D, is at a vantage point (e.g., on the ceiling) such that it can see all the three teammates, thus determining its own location and orientation. The four-robot formation theory and algorithms are presented, and experimental results with both simulated and real image data are provided to demonstrate the feasibility of this formation. Two 3D localization and control strategies are designed for applications such as search and rescue and surveillance in 3D urban environments where robots must be deployed in a full 3D space

In this paper we address the problem of fusing images from many video cameras or a moving video camera. The captured images have obvious motion parallax, but they will be aligned and integrated into a few mosaics with a large field-of-view (FOV) that preserve 3D information. We have developed a geometric representation that can re-organize the original perspective images into a set of parallel projections with different oblique viewing angles. In addition to providing a wide field of view, mosaics with various oblique views well represent occlusion regions that cannot be seen in a usual nadir view. Stereo pair(s) can be formed from a pair of mosaics with different oblique viewing angles and thus image-based 3D viewing can be achieved. This representation can be used as both an advanced video interface and a pre-processing step for 3D reconstruction.

In this paper, we present a new method for automatically and efficiently generating stereoscopic mosaics by seamless registration of images collected by a video camera mounted on an airborne platform. Using a parallel-perspective representation, a pair of geometrically registered stereo mosaics can be precisely constructed under quite general motion. A novel parallel ray interpolation for stereo mosaicing (PRISM) approach is proposed to make stereo mosaics seamless in the presence of obvious motion parallax and for rather arbitrary scenes. Parallel-perspective stereo mosaics generated with the PRISM method have better depth resolution than perspective stereo due to the adaptive baseline geometry. Moreover, unlike previous results showing that parallel-perspective stereo has a constant depth error, we conclude that the depth estimation error of stereo mosaics is in fact a linear function of the absolute depths of a scene. Experimental results on long video sequences are given.

As the threat of terrorism becomes more prevalent, the need for technological innovations for tighter security rises. Some of the areas where security needs to be increased are at border customs stations, airports, federal buildings, embassies, sporting events, and the like where checkpoints are being set up to inspect vehicles coming onto the premises. The proposed vehicle inspection system uses an at-ray of cameras and mosaicing techniques to generate different perspective views that provide a 3D representation of a vehicle's undercarriage. These mosaics allow a complete inspection of the undercarriage using a graphical user interface without requiring the use of inspection mirrors or the vehicle to drive over a manned inspection bay. The system is designed for simplicity so that operators will not require long training periods.

In this paper a compact representation, 3D Layered, Adaptive-resolution and Multi-perspective Panorama (LAMP), is proposed for representing large scale and 3D scenes with occlusion. Two kinds of 3D LAMP representations are constructed, i.e. the relief-like LAMP and the image-based LAMP, both of which concisely represent almost all the information from a long image sequence. The relief-like LAMP is basically a single extended multi-perspective panoramic view image with both texture and depth values, but each pixel has multiple values to represent results of occlusion recovery and resolution enhancement. The image-based LAMP, on the other hand, consists of a set of multi-perspective layers, each of which has both texture and depth maps, with adaptive time-sampling scales depending on depths of scene points. Several examples of 3D LAMP construction for real image sequences are given. The 3D LAMP is a concise and powerful representation for image-based rendering.

This paper presents a novel approach to stabilize the output of video camera installed on a moving vehicle in a rugged environment. A 2.5D interframe motion model is proposed so that the stabilization system can perform in situations where significant depth changes are present and the camera has both rotation and translation. Inertial motion filtering is proposed in order to eliminate the vibration of the video sequences with enhanced perceptual properties. The implementation of this new approach integrates four modules: pyramid-based motion detection, motion identification and 2.5D motion parameter estimation, inertial motion filtering, and affine-based motion compensation. The stabilization system can smooth unwanted vibrations or shakes of video sequences and achieve real-time speed. We test the system on IBM PC compatible machines and the experimental results show that our algorithm outperforms many algorithms which require parallel pipeline image processing machines.

This paper presents a novel approach for generating and analyzing epipolar plane images (EPIs) from video sequences taken from a moving platform subject to vibration so that the 3D model of an arbitrary scene can be constructed. Two problems are solved in our approach: (1) how to generate EPIs from video under a more general motion than a pure translation; (2) how to analyze the huge amount of data in the EPIs robustly and efficiently. For the first problem, a 3D image stabilization method is proposed which decouples the vibration from the vehicle's motion so that good EPIs and panoramic view images (PVIs) can be generated. For the second problem, we propose an efficient panoramic EPI analysis (PEPIA) method in which only one scanline of each EPI is processed. The PEPIA combines advantages of PVIs and EPIs and consists of three important steps: locus orientation detection, motion boundary localization, and occlusion/resolution recovery. The output of the PEPIA-a layered 3D panorama, is very useful in visual navigation and virtual reality modeling. Since camera calibration, image segmentation, feature extraction and matching are avoided, all the proposed algorithms are fully automatic and rather general. Results on real image sequences are given.

A new approach is presented to automatically build a dynamic and multi-resolution 360/spl deg/ panorama (DMP) from image sequences taken by a hand-held camera. A multi-resolution representation is built for the more interesting areas by means of camera zooming. The dynamic objects in the scene can be detected and represented separately. The DMP construction method is fast, robust and automatic, achieving 1 Hz in a 266 MHz PC.

This paper presents the results of integrating omnidirectional view image analysis and a set of adaptive backpropagation networks to understand the outdoor road scene by a mobile robot. Both the road orientations used for robot heading and the road categories used for robot localization are determined by the integrated system, the road understanding neural networks (RUNN). Classification is performed before orientation estimation so that the system can deal with road images with different types effectively and efficiently. An omni-view image (OVI) sensor captures images with 360 degree view around the robot in real-time. The rotation-invariant image features are extracted by a series of image transformations, and serve as the inputs of a road classification network (RCN). Each road category has its own road orientation network (RON), and the classification result (the road category) activates the corresponding RON to estimate the road orientation of the input image. Several design issues, including the network model, the selection of input data, the number of the hidden units, and learning problems are studied. The internal representations of the networks are carefully analyzed. Experimental results with real scene images show that the method is fast and robust.

A novel stereo matching algorithm has been proposed in this paper based on the observation that there are plenty of horizontal features in indoor environments. In this algorithm, the shape identity of all horizontal feature correspondences is achieved by means of a special image transformation-reprojection transformation. By using edge string as an intermediate structure, a generalized Hough transformation strategy fuses a variety of criteria, such as shape similarity disparity continuity and pixel attribution similarity, etc., together. In this way it copes with the correspondence of horizontal and non-horizontal features for 3D environment model building. First the principle of reprojection transformation using calibrated or uncalibrated images are discussed, then the generalized Hough transform strategy is addressed. Preliminary experimental results indicate that the algorithm is an effective and efficient method for indoor environment model building by using stereo methodology.

The authors present a novel approach using 2D spatio-temporal images for automatic traffic monitoring. A TV camera is mounted above the highway to monitor the traffic through two slice windows for each traffic lane. One slice window is along the lane and the other perpendicular to the lane axis. Two types of 2D spatio-temporal (ST) images are used in the system: the panoramic view image (PVI) and the epipolar plane image (EPI). The real-time vision system for automatic traffic monitoring, VISATRAM, an inexpensive system with a PC 486 and an image frame grabber has been tested with real road images. Not only can the system count the vehicles and estimate their speeds, but it can also classify the passing vehicles using 3D measurements (length, width and height). The VISATRAM works robustly under various light conditions including shadows in the day and vehicle lights at night, and automatically copes with the gradual and abrupt changes of the environment.

In this paper we model the problem of structure from motion as the range estimation with known motion. First, we approximate the motion within a reasonable time interval as a 3D translation and thus some image transformations are applied to convert an arbitrary motion to a 1D translation. Then we analyse the epipolar plane image in the Fourier domain to avoid the feature extraction and correspondence problems. Experimental results with real scene images have shown the efficiency and robustness of the approach.

Presents a new framework for detection of dynamic obstacles in the unstructured outdoor road environment by integrating binocular color image sequences. In the authors' system, color image segmentation, stereo obstacle detection, visual egomotion estimation, and moving object analysis, are all built-in task-oriented modules and hence are efficient and robust. Most of these functions can be performed in real-time. They are activated and integrated adaptively in the manner of neural parallel distributed processing (PDP). Experimental results are given to validate the authors' philosophy of so-called purposive vision, retinal mapping, adaptive integration and parallel distributed processing.>

Based on flat ground assumption, a correlation method is developed to estimate the orientation of a vehicle relative to its running environment. A weight function is introduced to account for figure variation caused by 3D dynamic scene and perspective effects caused by the camera system. The estimated parameters can be qualitatively used in visual navigation. Results indicate that robust estimation can be achieved in real time by using the powerful image processing system PIPE (Pipelined Image Processing Engine).>

A novel method called height from motion (HFM) is developed to estimate the motion and structure under planar motion. By using this method both translational and rotational motion (three degrees of freedom) can be treated in a unified manner. Based on the HFM method, the correspondence problem becomes easy to deal with, especially under translational motion. Experiments of real scene image sequences and the error analysis (theoretically and experimentally) have shown the efficiency and robustness of the method.>