Accurate electric load forecasting is of critical importance for modern power grids. It can help optimize energy management, reduce operational costs, and enhance grid stability. Existing load forecasting tools typically perform well when modelling long-term trends with substantive data upon which to build a model, but can perform poorly for short-term load forecasting or when data is sparse or incomplete. Diffusion models have recently emerged as powerful generative tools that excel in modelling complex distributions, making them a promising approach for electric load forecasting. In this paper, we build upon recent diffusion models for time series forecasting and explore the potential of combining diffusion models with prior models to improve performance. Additionally, we propose a metric-based meta-learning approach for fast data adaptation. Experimental results with this metric-based meta-learning approach on real-world load forecasting datasets outperform state-of-the-art baselines, showcasing the potential of diffusion-based refinement in practical forecasting applications.

Gesture-based communication is a standard underwater communication strategy that is taught to divers as part of their regular diver training and it would seem a natural mechanism to leverage for diver to robot communication underwater. Enabling an unmanned underwater vehicle (UUV) to understand such sequences would involve having the robot learn the large set of gestures that divers use and the way they are combined. As perfect transcription of gestures is unlikely, the communication process also requires an error-correcting framework to ensure that communication is clear and correct. Here we describe an interactive process that provides this infrastructure. A weakly supervised transfer learning approach is used to recognize standard SCUBA gestures in individual video frames and within a Sim2Real process to train a LSTM to recognize gesture sequences. This process is placed within a per-gesture and per-sequence interaction process to assist and confirm the recognition of individual gestures and to confirm entire gesture sequences. Individual aspects of this process and complete end-to-end operation are demonstrated using an unmanned underwater vehicle.

Speech emotion recognition (SER) is the task of automatically recognizing emotions expressed in spoken language. Current approaches focus on analyzing isolated speech segments to identify a speaker’s emotional state. Meanwhile, recent text-based emotion recognition methods have effectively shifted towards emotion recognition in conversation (ERC) that considers conversational context. Motivated by this shift, here we propose SERC-GCN, a method for speech emotion recognition in conversation (SERC) that predicts a speaker’s emotional state by incorporating conversational context, speaker interactions, and temporal dependencies between utterances. SERC-GCN is a two-stage method. First, emotional features of utterance-level speech signals are extracted. Then, these features are used to form conversation graphs that are used to train a graph convolutional network to perform SERC. We empirically evaluate the effectiveness of SERC-GCN and show that it outperforms the current state-of-the-art methods on the IEMOCAP benchmark dataset.

With the global aim of reducing carbon emissions, energy saving for communication systems has gained tremendous attention. Efficient energy-saving solutions are not only required to accommodate the fast growth in communication demand but solutions are also challenged by the complex nature of the load dynamics. Recent reinforcement learning (RL)-based methods have shown promising performance for network optimization problems, such as base station energy saving. However, a major limitation of these methods is the requirement of online exploration of potential solutions using a high-fidelity simulator or the need to perform exploration in a real-world environment. We circumvent this issue by proposing an offline reinforcement learning energy saving (ORES) framework that allows us to learn an efficient control policy using previously collected data. We first deploy a behavior energy-saving policy on base stations and generate a set of interaction experiences. Then, using a robust deep offline reinforcement learning algorithm, we learn an energy-saving control policy based on the collected experiences. Results from experiments conducted on a diverse collection of communication scenarios with different behavior policies showcase the effectiveness of the proposed energy-saving algorithms.

Traffic signal control (TSC) has seen substantial advancements through the application of reinforcement learning (RL) algorithms, which have shown remarkable potential in enhancing traffic flow efficiency. These RL-based approaches often surpass traditional rule-based methods, particularly in dynamic traffic environments. However, current RL solutions for TSC predominantly rely on model-free methods, necessitating extensive environmental interactions during training. This requirement can be prohibitively expensive or unfeasible in real-world implementations. Furthermore, existing methods have frequently neglected the issue of fairness in multi-intersection control, resulting in unbalanced congestion across different intersections. To address these challenges, we present FM2Light, a fairness-aware model-based multi-agent RL framework for TSC. Our approach leverages an ensemble of global world models for generating synthetic samples to enhance sample efficiency, thereby mitigating the data-intensive nature of the training process. Additionally, FM2Light incorporates a refined reward structure to promote fairness and improve coordination across multiple intersections. Extensive evaluations conducted in diverse real-world scenarios demonstrate that FM2Light achieves performance comparable to or exceeding that of model-free RL (MFRL) methods, while significantly reducing sample requirements and ensuring more equitable control among multiple agents.

Science fiction has long promised a future within which robots assist humans in many facets of their daily lives, and robot technology is advancing at a pace which suggests that the necessary technology already exists, or may exist, in the near future. But, once the technology is in place, how accepting will humans be to autonomous machines performing tasks traditionally performed by humans? Are we designing and developing robots that are human centric? In a study involving 357 undergraduate students, we found that acceptance of robots was dependent upon previous exposure to different forms of technology (i.e., robots, avatars, video games). Men were more likely to have previous exposure to technology, and were therefore more likely to accept robots and avatars in different tasks compared to women. Enhancing the acceptability of robots by both men and women will require an increased exposure to technology, and women may require additional experience with technology to close the technology acceptance gap.

With the increasing use of data-intensive mobile applications and the number of mobile users, the demand for wireless data services has been increasing exponentially in recent years. In order to address this demand, a large number of new cellular base stations are being deployed around the world, leading to a significant increase in energy consumption and greenhouse gas emission. Consequently, energy consumption has emerged as a key concern in the fifth-generation (5G) network era and beyond. Reinforcement learning (RL), which aims to learn a control policy via interacting with the environment, has been shown to be effective in addressing network optimization problems. However, for reinforcement learning, especially deep reinforcement learning, a large number of interactions with the environment are required. This often limits its applicability in the real world. In this work, to better deal with dynamic traffic scenarios and improve real-world applicability, we propose a transfer deep reinforcement learning framework for energy optimization in cellular communication networks. Specifically, we first pre-train a set of RL-based energy-saving policies on source base stations and then transfer the most suitable policy to the given target base station in an unsupervised learning manner. Experimental results demonstrate that base station energy consumption can be reduced significantly using this approach.

The association of mobile devices with network resources (e.g., base stations, frequency bands/channels), known as load balancing, is critical to reduce communication traffic congestion and network performance. Reinforcement learning (RL) has shown to be effective for communication load balancing and achieves better performance than currently used rule-based methods, especially when the traffic load changes quickly. However, RL-based methods usually need to interact with the environment for a large number of time steps to learn an effective policy and can be difficult to tune. In this work, we aim to improve the data efficiency of RL-based solutions to make them more suitable and applicable for real-world applications. Specifically, we propose a simple, yet efficient and effective deep RL-based wireless network load balancing framework. In this solution, a set of good initialization values for control actions are selected with some cost-efficient approach to center the training of the RL agent. Then, a deep RL-based agent is trained to find offsets from the initialization values that optimize the load balancing problem. Experimental evaluation on a set of dynamic traffic scenarios demonstrates the effectiveness and efficiency of the proposed method.

With the continuous growth in communication network complexity and traffic volume, communication load balancing solutions are receiving increasing attention. Specifically, reinforcement learning (RL)-based methods have shown impressive performance compared with traditional rule-based methods. However, standard RL methods generally require an enormous amount of data to train, and generalize poorly to scenarios that are not encountered during training. We propose a policy reuse framework in which a policy selector chooses the most suitable pre-trained RL policy to execute based on the current traffic condition. Our method hinges on a policy bank composed of policies trained on a diverse set of traffic scenarios. When deploying to an unknown traffic scenario, we select a policy from the policy bank based on the similarity between the previous-day traffic of the current scenario and the traffic observed during training. Experiments demonstrate that this framework can outperform classical and adaptive rule-based methods by a large margin.

Our work examines the way in which large language models can be used for robotic planning and sampling in the context of automated photographic documentation. Specifically, we illustrate how to produce a photo-taking robot with an exceptional level of semantic awareness by leveraging recent advances in general purpose language (LM) and vision-language (VLM) models. Given a high-level description of an event we use an LM to generate a natural-language list of photo descriptions that one would expect a photographer to capture at the event. We then use a VLM to identify the best matches to these descriptions in the robot's video stream. The photo portfolios generated by our method are consistently rated as more appropriate to the event by human evaluators than those generated by existing methods.

The underwater environment provides a range of interesting applications for human-robot teams. A critical issue for such teams is the development of an appropriate communication mechanism between humans and robots operating at depth. Humans operating at depth have developed an applied gesture-based communication language that can be leveraged to enable this communication, but it would be expensive and perhaps impractical to develop a hand-labelled dataset of these gestures to support a machine learning-based approach to the task. To avoid the cost of hand labelling such a large dataset, here we automate the process of collecting a labelled dataset through the use of a simple model trained on a hand-labelled dataset that only identifies salient objects (divers, their heads and hands), and then use a weakly supervised learning process to label a complex set of diver gestures. The result of this process is a system that can recognize a large number of diver hand gestures. Performance of the resulting system is compared against a hand-labelled set of diver gestures.

The modern power system is transitioning towards increasing penetration of renewable energy generation and demand from different types of electrical appliances. With this transition, residential load forecasting, especially short-term load forecasting (STLF), is becoming more and more challenging and important. Accurate short-term load forecasting can help improve energy dispatching efficiency and, as a consequence, reduce overall power system operation cost. Most current load forecasting algorithms assume that there is a large amount of training data available upon which to learn a reliable load forecasting model. However, this assumption can be challenging for real-world applications. In this work, we first propose the use of transfer learning and an attention mechanism to improve short-term load forecasting for a target domain with only a limited amount of available data. Furthermore, we extend the proposed method to utilize heterogeneous features which enables the approach to deal with more complex scenarios in the real world. Experimental results using real-world data sets show that the proposed methods can improve forecasting accuracy by a large margin over several existing baselines.

With the increasing adoption of renewable energy generation and electric devices, electric load forecasting, especially short-term load forecasting (STLF), is becoming more and more important. The widespread adoption of smart meters makes it possible to utilize complex machine learning models for both aggregated load and single-home residential load forecasting. Similar homes in nearby locations are likely to have similar load consumption patterns and this similarity can be used to improve the overall forecasting performance. However, most current work on load forecasting focuses on single learning task without exploiting the benefit of joint learning. In this paper, we propose the use of the multi-task learning (MTL) framework with long short-term memory (LSTM) recurrent neural networks for both aggregated and single home STLF. We propose a MTL-based forecasting algorithm for aggregated load forecasting in which single home forecasting is formulated as a single learning task within the MTL framework. This algorithm is extended for single home load forecasting in which load forecasting for a particular home becomes the primary learning task. Experimental results on real-world data sets demonstrate that residential load forecasting for both aggregated load and a single home can be improved within the MTL framework.

With the increasing popularity of electric vehicles and the growing trend of working from home, electricity consumption in the residential sector is expected to continue to grow rapidly over the next few years. As a consequence, short-term residential load forecasting is becoming even more vital for the reliability and sustainability of the smart grid. Although deep learning models have shown impressive success in different areas including short-term electric load forecasting, such models require a large amount of training data. For many real-world load forecasting cases, we may not have enough training data to learn a reliable forecasting model. In this paper, we address this challenge through the use of boosting-based transfer learning with multiple sources. We first train a set of deep regression models on source houses that can provide relatively abundant data. We then transfer these learned models via the boosting framework to support data-scarce target houses. The transfer process is selective and customized for each target house to minimize the potential for negative transfer. Experimental results, based on real-world residential data sets, show that the proposed method can significantly improve forecasting accuracy.

Manipulating objects with dexterity requires timely feedback that simultaneously leverages the senses of vision and touch. In this paper, we focus on the problem setting where both visual and tactile sensors provide pixel-level feedback for Visuotactile reinforcement learning agents. We investigate the challenges associated with multimodal learning and propose several improvements to existing RL methods; including tactile gating, tactile data augmentation, and visual degradation. When compared with visual-only and tactile-only baselines, our Visuotactile-RL agents showcase (1) significant improvements in contact-rich tasks; (2) improved robustness to visual changes (lighting/camera view) in the workspace; and (3) resilience to physical changes in the task environment (weight/friction of objects).

In this paper, we present the Sample Efficient Social Navigation from Observation (SESNO) algorithm that efficiently learns socially-compliant navigation policies from observations of human trajectories. SESNO is an inverse reinforcement learning (IRL)-based algorithm that learns from human trajectory observations without knowledge of their actions. We improve the sample-efficiency over previous IRL-based methods by introducing a shared experience replay buffer that allows reuse of past trajectory experiences to estimate the policy and the reward. We evaluate SESNO using publicly available pedestrian motion data sets and compare its performance to related baseline methods in the literature. We show that SESNO yields performance superior to existing baselines while dramatically improving the sample complexity by using as few as a hundredth of the samples required by existing baselines.

Predicting the future interaction of objects when they come into contact with their environment is key for autonomous agents to take intelligent and anticipatory actions. This paper presents a perception framework that fuses visual and tactile feedback to make predictions about the expected motion of objects in dynamic scenes. Visual information captures object properties such as 3D shape and location, while tactile information provides critical cues about interaction forces and resulting object motion when it makes contact with the environment. Utilizing a novel See-Through-your-Skin (STS) sensor that provides high resolution multimodal sensing of contact surfaces, our system captures both the visual appearance and the tactile properties of objects. We interpret the dual stream signals from the sensor using a Multimodal Variational Autoencoder (MVAE), allowing us to capture both modalities of contacting objects and to develop a mapping from visual to tactile interaction and vice-versa. Additionally, the perceptual system can be used to infer the outcome of future physical interactions, which we validate through simulated and real-world experiments in which the resting state of an object is predicted from given initial conditions.

Despite its importance in the real-world, manual (hand) dexterity is often ignored in medical-based virtual training environments that have traditionally focused on cognitive and affective skills development. Psychomotor (technical) skills, particularly those related to manual dexterity, are fundamental to various medical procedures and ignoring them in virtual based training tools can lead to a sub-optimal training experience. Here, we present a novel, consumer-level data glove that provides accurate user interactions involving the proximal and medial phalanges, interactions that are relevant in many manual dexterity tasks. We also outline how this novel data glove is being incorporated into an existing serious gaming platform for anesthesia training that currently focuses on cognitive and affective skills development only. The addition of psychomotor skills development through the adoption of simulated tactile feedback will provide a more complete serious gaming training platform.

Within a cellular network, load balancing between different cells is of critical importance to network performance and quality of service. Most existing load balancing algorithms are manually designed and tuned rule-based methods where near-optimality is almost impossible to achieve. These rule-based meth-ods are difficult to adapt quickly to traffic changes in real-world environments. Given the success of Reinforcement Learning (RL) algorithms in many application domains, there have been a number of efforts to tackle load balancing for communication systems using RL-based methods. To our knowledge, none of these efforts have addressed the need for data efficiency within the RL framework, which is one of the main obstacles in applying RL to wireless network load balancing. In this paper, we formulate the communication load balancing problem as a Markov Decision Process and propose a data-efficient transfer deep reinforcement learning algorithm to address it. Experimental results show that the proposed method can significantly improve the system performance over other baselines and is more robust to environmental changes.

Although existing cellular network base stations are typically immobile, the recent development of small form factor base stations and self driving cars has enabled the possibility of deploying a team of continuously moving base stations that can reorganize the network infrastructure to adapt to changing network traffic usage patterns. Given such a system of mobile base stations (MBSes) that can freely move on the road, how should their path be planned in an effort to optimize the experience of the users? This paper addresses this question by modeling the problem as a Markov Decision Process where the actions correspond to the MBSes deciding which direction to go at traffic intersections; states corresponds to the position of MBSes; and rewards correspond to minimization of packet loss in the network. A Monte Carlo Tree Search (MCTS)-based anytime algorithm that produces path plans for multiple base stations while optimizing expected packet loss is proposed. Simulated experiments in the city of Verdun, QC, Canada with varying user equipment (UE) densities and random initial conditions show that the proposed approach consistently outperforms myopic planners, and is able to achieve near-optimal performance.

We introduce a new class of vision-based sensor and associated algorithmic processes that combine visual imaging with high-resolution tactile sending, all in a uniform hardware and computational architecture. We demonstrate the sensor's efficacy for both multi-modal object recognition and metrology. Object recognition is typically formulated as an unimodal task, but by combining two sensor modalities we show that we can achieve several significant performance improvements. This sensor, named the See-Through-your-Skin sensor (STS), is designed to provide rich multi-modal sensing of contact surfaces. Inspired by recent developments in optical tactile sensing technology, we address a key missing feature of these sensors: the ability to capture a visual perspective of the region beyond the contact surface. Whereas optical tactile sensors are typically opaque, we present a sensor with a semitransparent skin that has the dual capabilities of acting as a tactile sensor and/or as a visual camera depending on its internal lighting conditions. This paper details the design of the sensor, showcases its dual sensing capabilities, and presents a deep learning architecture that fuses vision and touch. We validate the ability of the sensor to classify household objects, recognize fine textures, and infer their physical properties both through numerical simulations and experiments with a smart countertop prototype.

Current methods for human robot interaction in the underwater domain seem antiquated in comparison to their terrestrial counterparts. Visual tags and custom built wired remotes are commonplace underwater, but such approaches have numerous drawbacks. Here we describe a method for human robot interaction underwater that borrows from the long standing history of diver communication using hand signals; a three stage approach for diver-robot communication using a series of neural networks.

Robot-diver communication underwater is complicated by the attenuation of RF signals, the complexities of the environment in terms of deploying interaction devices, and issues related to the cognitive loading of human operators. Humans operating underwater have developed a simple yet effective strategy for diver-diver communication based on the visual recognition of gestures. Can a similar approach be effective for diver-robot communication? Here we present experiments with SCUBANet, an underwater detection dataset of body parts associated with diver-robot communication. Given the nature of standard diver gestures, here we concentrate on diver recognition and in particular on diver body-head-hand localization and examine the feasibility of using a CNN-based approach to address this problem. Such data-driven approaches typically require an appropriately annotated dataset. The SCUBANet dataset contains images of object classes commonly encountered during human-robot communication underwater. Object classes are labeled using per-instance bounding boxes. Annotations were created through crowd sourcing via a web-based interface to ease deployment. We provide baseline performance on diver and diver component recognition and localization using transfer learning on three widely available pre-trained models.

Trails typically lack standard markers that characterize roadways. Nevertheless, trails are useful for off-road navigation. Here, trail following problem is approached by identifying the deviation of the robot from the heading direction of the trail by fine-tuning a pre-trained Inception-V3 [1] network. Key questions considered in this work include the required number, nature and geometry of the cameras and how trail types  – encoded in pre-existing maps  – can be exploited in addressing this task. Through evaluation of representative image datasets and on-robot testing we found: (i) that although a single camera cannot estimate angular deviation from the heading direction, but it can reliably detect that the robot is, or is not, following the trail; (ii) that two cameras pointing towards the left and the right can be used to estimate heading reliably within a differential framework; (iii) that trail nature is a useful tool for training networks for different trail types.

Simultaneous Localization and Mapping (SLAM) is a key stepping stone on the road to truly autonomous robots. SLAM is of particular importance to robots with large motion estimation problems, such as robots operating on the surface of aquatic GPS-denied environments where a paucity of local landmarks complicates SLAM and accurate navigation. Visual sensors have proven to be an effective tool for SLAM generally and have wide applicability, but is vision enough to solve SLAM in this environment, and how important are other sensors including a compass and water column depth to solve SLAM for an aquatic surface vehicle? Here we show that more sensors are almost always helpful in terms of improving SLAM performance in such a situation but that a compass is a particularly useful sensor for SLAM for autonomous surface vehicles; suggesting that a compass is a worthwhile investment for such a robot, and that compass alternatives should be considered when operating an autonomous vehicle in environments that are both GPS and compass-denied.

The development of effective user interfaces for an autonomous system can be quite difficult, especially for devices that are to be operated in the field where access to standard computer platforms may be difficult or impossible. One approach in this type of environment is to utilize tablet or phone devices, which when coupled with an appropriate tool such as ROSBridge can be used to connect with standard robot middleware. This has proven to be a successful approach for devices with mature user interface requirements but may require significant software development for experimental systems. Here we describe RCON, a software tool that allows user interfaces on iOS devices to be configured on the device itself, in real time, in response to changes in the robot software infrastructure or the needs of the operator. The system is described in detail along with the accompanying communication framework and the process of building a user interface for a simple autonomous device.

Although many commercially available robots ship with a version of ROS this is not as true for many external sensors. There is a lack of ROS support for many devices and sensors one might use to extend the capabilities of a robot. As robots are deployed in more complex environments there is the need for more specialized sensors. In particular in the aquatic domain there is the need for support for depth sounders. This paper describes the design and construction process for building a ROS node for a NMEA 0183 compliant depth and temperature transducer and a strategy for extending this design to other NMEA devices.

Sampling-based algorithms are a common approach to high-dimensional real-world path planning problems. Unfortunately the solutions found using such planners are often not practical in that they do not take into account soft application-specific constraints. This paper formulates the practicality of paths based on the notion of soft constraints found in the Planning Domain Definition Language 3 (PDDL3) (Gerevini and Long, 2005) and a range of optimization strategies are developed targeted towards user-preferred qualities by integrating soft constraints in the pre-processing, planning and post-processing phases of the sampling-based path planners. An auction-based resource allocation approach coordinates competing optimization strategies. This approach uses an adaptive bidding strategy for each optimizer and in each round the optimizer with the best predicted performance is selected. This general coordination system allows for flexibility in both the number and types of the optimizers used. Experimental validation demonstrates the effectiveness of the approach.

Pose estimation is an important capability for mobile agents. A wide variety of solutions have been proposed, but work in the literature has focused primarily on solutions for robots whose mobility is restricted to the ground plane. In this work we present a framework for 6DOF pose estimation. Normally the increased computational cost associated with this higher dimensional space makes pose estimation intractable. The approach presented here addresses the computational issues associated with the higher dimensional problem by decoupling orientation estimation from position estimation. Assuming that orientation can be estimated separately from position allows efficient methods to be used for the (unimodal) orientation estimate, while more sophisticated methods are used for the position estimate. Although similar to Rao-Blackwellization, the approach is essentially reversed. Results on real and simulated datasets and a comparison with a naive 6DOF filter are presented.

The fundamental problem in robotic exploration and mapping of an unknown environment is answering the question 'have I been here before?', which involves disambiguating the robot's current location from previously visited or known locations. One approach to answering this problem in embedded topological worlds is to resort to the use of an external aid that can help the robot disambiguate places. Here we investigate the power of different marker-based aids in exploring undirected topological graphs. We demonstrate that for undirected graphs, certain marker aids are insufficient, while others have powers that are sufficient to develop asymptotically optimal exploration algorithms.

Human-robot communication is a complex problem even in the terrestrial domain. Failure to properly communicate instructions to a robot and receive appropriate feedback can at the very least hamper the ability of the robot to perform its task, and at worst prevent the task from being completed. The problem of providing effective communication between a robot and its operator becomes even more complex underwater. Many communication channels available in the terrestrial domain become unavailable, and communication between team members and task oversight become even more complex. This paper describes initial experiments with the AQUATablet - a robot interaction device designed to be operated by a diver tethered to, or in visual communication with, an underwater robot. The basic requirements of the device are described along with design considerations and results of initial experiments with the device conducted in the pool and in the open ocean.

Literature and folklore is rife with a range of oracles that have been used by explorers to explore unknown environments. But how effective are these various oracles? This paper considers the power of string and string-like oracles to map an unknown embedded topological environment. We demonstrate that for undirected graphs, even very short strings can be used to explore an unknown environment but that significant performance improvements can be found when longer strings are available.

The ISO 9241 Part 9 standard pointing task is used to evaluate passive haptic feedback in target selection in a virtual environment (VE). Participants performed a tapping task using a tracked stylus in a CAVE both with, and without passive haptic feedback provided by a plastic panel co-located with the targets. Pointing throughput (but not speed nor accuracy alone) was significantly higher with haptic feedback than without it, confirming previous results using an alternative experimental paradigm.

Virtual reality has been proposed as a training regime for a large number of tasks from surgery rehearsal (cf. [Robb et al. 1996], to combat simulation (cf. [U. S. Congress, Office of Technology Assessment 1994]) to assiting in basic design (cf. [Fa et al. 1992]). Virtual reality provides a novel and effective training medium for applications in which training "in the real world" is dangerous or expensive. Here we describe the C2SM simulator system -- a virtual reality-based training system that provides an accurate simulation of the CBRNE Crime Scene Modeller System (see [Topol et al. 2008]). The training system provides a simulation of both the underlying robotic platform and the C2SM sensor suite, and allows training of the system to take place without physically deploying the robot or the simulation of chemical and radiological agents that might be present. This paper describes the basic structure of the C2SM simulator and the software components that were used to construct it.

The development of sensors capable of obtaining 3D scans of crime scenes is revolutionizing the ways in which crime scenes can be analyzed and at the same time is driving the need for the development of sophisticated tools to represent and store this data. Here we describe the design of a multimedia database suitable for representing and reasoning about crime scene data. The representation is grounded in the physical environment that makes up the crime scene and provides mechanisms for representing both traditional (forms-based) data as well as 3D scan and other complex spatial data.

The underwater environment presents many challenges for robotic sensing including highly variable lighting, the presence of dynamic objects, and the six degree of freedom (6DOF) 3D environment. Yet in spite of these challenges the aquatic environment presents many real and practical applications for robotic sensors. A common requirement of many of these tasks is the need to construct accurate 3D representations of structures in the environment. In order to address this requirement we have developed a stereo vision-inertial sensing device that we have successfully deployed to reconstruct complex 3D structures in both the aquatic and terrestrial domains. The sensor temporally combines 3D information, obtained using stereo vision algorithms with a 3DOF inertial sensor. The resulting point cloud model is then converted to a volumetric representation and a textured polygonal mesh is extracted for later processing. Recently obtained underwater reconstructions of wrecks and coral obtained with the sensor are presented.

In vision-based autonomous spacecraft docking multiple views of scene structure captured with the same camera and scene geometry is available under different lighting conditions. These "multiple-exposure" images must be processed to localize visual features to compute the pose of the target object. This paper describes a robust multi-channel edge detection algorithm that localizes the structure of the target object from the local gradient distribution computed over these multiple-exposure images. This approach reduces the effect of the illumination variation including the effect of shadow edges over the use of a single image. Experiments demonstrate that this approach has a lower false detection rate than the average response of the Canny edge detector applied to the individual images separately.

Modeling the acoustics of an environment is a complex and challenging task. Here we describe the sonel mapping approach to acoustical rendering. Sonel mapping is a Monte-Carlo-based approach to modeling diffuse, specular, absorption and diffraction effects in an efficient manner. The approach models many of the subtle interaction effects required for realistic acoustical modeling, and the approach is computationally efficient allowing it to be used to acoustically model interactive virtual environments

The aquatic realm is ideal for testing autonomous robotic technology. The challenges presented in this environment are numerous due to the highly dynamic nature of the medium. Applications for underwater robotics include the autonomous inspection of coral reef, ships, pipelines, and other environmental assessment programs. In this paper we present current results in using 6DOF entropy minimization SLAM (simultaneous localization and mapping) for creating dense 3D visual maps of underwater environments that are suitable for such applications. The proposed SLAM algorithm exploits dense information coming from a stereo system, and performs robust egomotion estimation and global-rectification following an optimization approach

Guiding the spacecraft docking process requires the use of sensors that estimate the relative position of the two vessels. This task is complicated by the widely variable on-orbit illumination. To combat this, controllable docking cameras are augmented by computer-controlled illuminants. But how should these illumination and capture parameters be controlled and how should the images obtained under different conditions be combined in order to estimate the relative pose of the vessels? We address these issues in the "Lights and Camera" system. Images captured with the same camera and scene geometry but under different lighting conditions are merged, and the resulting edges are used to estimate the target’s pose. A high level controller monitors the imaging process and determines the set of images to capture and use for pose estimation. This paper describes the "Lights and Camera" system architecture and initial results of its operation on mockups of space hardware.

Coral reef health is an indicator of global climate change and coral reefs themselves are important for sheltering fish and other aquatic life. Monitoring reefs is a time-consuming and potentially dangerous task and as a consequence autonomous robotic mapping and surveillance is desired. This paper describes an underwater vision-based sensor to aid in this task. Underwater environments present many challenges for vision-based sensors and robotic vehicles. Lighting is highly variable, optical snow/particulate matter can confound traditional noise models, the environment lacks visual structure, and limited communication between autonomous agents including divers and surface support exacerbates the potentially dangerous environment. We describe experiments with our multi-camera stereo reconstruction algorithm geared towards coral reef monitoring. The sensor is used to estimate volumetric scene structure while simultaneously estimating sensor ego-motion. Preliminary field trials indicate the utility of the sensor for 3D reef monitoring and results of land-based evaluation of the sensor are shown to evaluate the accuracy of the system

We describe recent results obtained with AQUA, a mobile robot capable of swimming, walking and amphibious operation. Designed to rely primarily on visual sensors, the AQUA robot uses vision to navigate underwater using servo-based guidance, and also to obtain high-resolution range scans of its local environment. This paper describes some of the pragmatic and logistic obstacles encountered, and provides an overview of some of the basic capabilities of the vehicle and its associated sensors. Moreover, this paper presents the first ever amphibious transition from walking to swimming.

This paper describes an underwater walking robotic system being developed under the name AQUA, the goals of the AQUA project, the overall hardware and software design, the basic hardware and sensor packages that have been developed, and some initial experiments. The robot is based on the RHex hexapod robot and uses a suite of sensing technologies, primarily based on computer vision and INS, to allow it to navigate and map clear shallow-water environments. The sensor-based navigation and mapping algorithms are based on the use of both artificial floating visual and acoustic landmarks as well as on naturally occurring underwater landmarks and trinocular stereo.

Virtual reality displays introduce spatial distortions that are very hard to correct because of the difficulty of precisely modelling the camera from the nodal point of each eye. How significant are these distortions for spatial perception in virtual reality? In this study, we used a helmet-mounted display and a mechanical head tracker to investigate the tolerance to errors between head motions and the resulting visual display. The relationship between the head movement and the associated updating of the visual display was adjusted by subjects until the image was judged as stable relative to the world. Both rotational and translational movements were tested, and the relationship between the movements and the direction of gravity was varied systematically. Typically, for the display to be judged as stable, subjects needed the visual world to be moved in the opposite direction to the head movement by an amount greater than the head movement itself, during both rotational and translational head movements, although a large range of movement was tolerated and judged as appearing stable. These results suggest that it not necessary to model the visual geometry accurately and suggest circumstances when tracker drift can be corrected by jumps in the display which will pass unnoticed by the user.

We present a robust and portable visual-based skin and face detection system developed for use in a multiple speaker teleconferencing system, employing both audio and video cues. An omni-directional video sensor is used to provide a view of the entire visual hemisphere, thereby allowing for multiple dynamic views of all the participants. Regions of skin are detected using simple statistical methods, along with histogram color models for both skin and non-skin color classes. Regions of skin belonging to the same person are grouped together, and using simple spatial properties, the position of each person's face is inferred. Preliminary results suggest the system is capable of detecting human faces present in an omni-directional image despite the poor resolution inherent with such an omni-directional sensor.

To enhance presence, facilitate sensory motor performance, and avoid disorientation or nausea, virtual-reality applications require the perception of a stable environment. End-end tracking latency (display lag) degrades this illusion of stability and has been identified as a major fault of existing virtual-environment systems. Oscillopsia refers to the perception that the visual world appears to swim about or oscillate in space and is a manifestation of this loss of perceptual stability of the environment. The effects of end-end latency and head velocity on perceptual stability in a virtual environment were investigated psychophysically. Subjects became significantly more likely to report oscillopsia during head movements when end-end latency or head velocity were increased. It is concluded that perceptual instability of the world arises with increased head motion and increased display lag. Oscillopsia is expected to be more apparent in tasks requiring real locomotion or rapid head movement.

Multiple mobile robots, or robot collectives, have been proposed as solutions to various tasks in which distributed sensing and action are required. Here we consider applying a collective of robots to the paparazzi problem - the problem of providing sensor coverage of a target robot. We demonstrate how the computational task of the collective can be formulated as a global energy minimization task over the entire collective, and show how individual members of the collective can solve the task in a distributed fashion so that the entire collective meets its goal. This result is then extended to consider unbounded communication delays between members and complete failure of individual members of the collective.

Although technologies such as head-mounted displays and CAVEs can be used to provide large immersive visual displays within small physical spaces, it is difficult to provide virtual environments which are as large physically as they are visually. A fundamental problem is that tracking technologies which work well in a small enclosed environment do not function well over longer distances. In this paper, we describe Trike-a 'rideable' computer system which can be used to generate and explore large virtual spaces both visually and physically. This paper describes the hardware and software components of the system and a set of experiments which have been performed to investigate how the different perceptual cues that can be provided with Trike interact within an immersive environment.

The design of virtual environments usually concentrates on constructing a realistic visual simulation and ignores the non-visual cues normally associated with moving through an environment. The lack of the normal complement of cues may contribute to cybersickness and may affect operator performance. Previously (1998) we described the effect of adding vestibular cues during passive linear motion and showed an unexpected dominance of the vestibular cue in determining the magnitude of the perceived motion. Here we vary the relative magnitude of the visual and vestibular cues and describe a simple linear summation model that predicts the resulting perceived magnitude of motion. The model suggests that designers of virtual reality displays should add vestibular information in a ratio of one to four with the visual motion to obtain convincing and accurate performance.

One of the challenges facing computer vision systems used in space is the presence of specular surfaces. Such surfaces lead to several adverse effects such as the creation of reflected "virtual" images of objects due to specular reflections. These effects may lead to incorrect measurements and loss of data in the case of sensor saturation or inadequate intensity of the returned laser beams in the case of an active illuminant. In addition, the instruments inside space structures such as satellites may be extremely sensitive to active illuminants such as laser beams or radar signals, and thus passive vision systems which rely on either natural or low-power projection systems are preferred over active sensing technologies. Here we consider the task of recovering the local surface structure of highly specular surfaces such as satellites using passive stereopsis without resulting to the introduction of additional light source.

By capitalizing on the known properties of harmonic potential functions this work develops a new approach to probability-based path planning that is intuitive, free from local traps (local minima) and computationally less complex than many existing methods. Although the approach presented here is based on the hill-climbing method, it is still able to guarantee goal attainment. Furthermore the algorithm presented here is able to handle arbitrary-shaped geometries and does not require any geometrical or topological approximation at the environment representation level.

This paper deals with coordinating behaviour in a multi-autonomous robot system. When two or more autonomous robots must interact in order to accomplish some common goal, communication between the robots is essential. Different inter-robot communications strategies give rise to different overall system performance and reliability. After a brief consideration of some theoretical approaches to multiple robot collections, we present concrete implementations of different strategies for convoy-like behaviour. The convoy system is based around two RWI B12 mobile robots and uses only passive visual sensing for inter-robot communication. The issues related to different communication strategies are considered.

When a robot moves about a 2D world such as a planar surface, it is important that obstacles to the robot's motions be detected. This classical problem of has proven to be difficult. Many researchers have formulated this problem as being the process of determining where a robot cannot move due to the presence of obstacles. An alternative approach presented here is to determine where an robot can go by identifying floor regions for which the planar floor assumption can be verified. A stereo vision system is developed for Floor Anomaly Detection (FAD), and its relationship to existing stereo obstacle detection algorithms is described.

When a particular point is fixated by an active stereo system different portions of the world are brought into interocular alignment. This region is known as the horoptor. Through an examination of the horoptor under different viewing conditions it is demonstrated that for certain binocular tasks it is desirable to manipulate the horoptor by rotating (torquing) the cameras about their optical axes. This manipulation can be passive for operations such as stereo based obstacle detection for mobile robots, or active for active binocular heads. Techniques for both situations are presented.>

When a particular 3D point is fixated by a robotic stereo system different portions of the world are brought into interocular alignment. This region is known as the horoptor. Purposeful modifications to the binocular geometry can be used to bring different regions of three-space closer to the horoptor: camera pan and tilt define the rough structure of the horoptor, while camera torsion can be used to change its local shape. Theoretical and empirical results suggest that for binocular vision tasks: 1) it is important to understand the region of three space that contains the horoptor curve; and 2) it is possible to control this shape in an active way so as to simplify certain binocular tasks.

This paper describes research on the ARK (Autonomous Mobile Robot in a Known Environment) project. The technical objective of the project is to build a robot that can navigate and carry out survey/inspection tasks in a complex but known industrial environment. Rather than altering the robots environment by adding easily identifiable beacons the robot relies on naturally occurring objects to use as visual landmarks for navigation. The robot is equipped with various sensors that are used to detect unmapped obstacles, landmarks and objects. This paper describes the robot's industrial environment, it's control architecture, and some results in processing the robot's range and vision sensor data for navigation.>

In many cases several mobile robots (autonomous agents) can be used together to accomplish tasks that would be either more difficult or impossible for a robot acting alone. Many different models have been suggested for the makeup of such collections of robots. In this paper the authors present a taxonomy of the different ways in which such a collection of autonomous robotic agents can be structured. It is shown that certain swarms provide little or no advantage over having a single robot, while other swarms can obtain better than linear speedup over a single robot. There exist both trivial and non-trivial problems for which a swarm of robots can succeed where a single robot will fail. Swarms are more than just networks of independent processors - they are potentially reconfigurable networks of communicating agents capable of coordinated sensing and interaction with the environment.

ARK (Autonomous Robot for a Known environment), is a visually-guided mobile robot which is being constructed as part of the Precarn project in mobile robotics. ARK operates in a previously mapped environment and navigates with respect to visual landmarks that have been previously located. While the robot moves, it utilizes an active vision sensor to register the robot with respect to these landmarks. As the landmarks may be scarce in certain regions of its environment, ARK plans paths which minimize both path length and path uncertainty. The global path planner assumes that the robot will use a Kalman filter to integrate landmark information with odometry data to correct path deviations as the robot moves, and then uses this information to choose a path which reduces the expected path deviation.

We present the design of a controllable stereo vision head. TRISH (The Toronto IRIS Stereo Head) is a binocular camera mount, consisting of two fixed focal length color cameras with automatic gain control forming a verging stereo pair. TRISH is capable of version (rotation of the eyes about the vertical axis so as to maintain a constant disparity), vergence (rotation of each eye about the vertical axis so as to change the disparity), pan (rotation of the entire head about the vertical axis), and tilt (rotation of each eye about the horizontal axis). One novel characteristic of the design is that each camera can rotate about its own optical axis (torsion). Torsional movement makes it possible to minimize the vertical component of the two-dimensional search which is associated with stereo processing in verging stereo systems.

An algorithm is presented for the smooth tracking of a target in three-dimensional space by a binocular head which is capable of vergence, version, and tilt eye movements. This algorithm utilizes stereomotion channels to obtain a measurement of the three-dimensional velocity of the target, and then uses this velocity within a control loop to keep the target center at the fixation point of the binocular head. Although stereomotion alone is insufficient to accurately drive binocular eye movements, relative stereomotion is a useful measurement and could be easily integrated into a positional error driven tracking system.>

Addressed is the problem of robotic exploration of a graphlike world, where no distance or orientation metric is assumed of the world. The robot is assumed to be able to autonomously traverse graph edges, recognize when it has reached a vertex, and enumerate edges incident upon the current vertex relative to the edge via which it entered the current vertex. The robot cannot measure distances, and it does not have a compass. It is demonstrated that this exploration problem is unsolvable in general without markers, and, to solve it, the robot is equipped with one or more distinct markers that can be put down or picked up at will and that can be recognized by the robot if they are at the same vertex as the robot. An exploration algorithm is developed and proven correct. Its performance is shown on several example worlds, and heuristics for improving its performance are discussed.>

Previous work has demonstrated that the task of recovering local disparity measurements can be reduced to the task of measuring the local phase between bandpass signals extracted from the left and right cameras. In computing this local phase difference, earlier algorithms expressed the computational task as a nonlinear differential equation to be solved at each image point. Although this approach has great appeal as a model for biological disparity measurement, the solving of a differential equation at a large number of image points and disparities makes the algorithm unsuitable for serial digital computer applications. Here, the authors demonstrate how the approach of recovering disparity from the measurement of local phase differences can be accomplished without the computational expense exhibited by previous algorithms. This disparity measurement technique is embedded within a simple coarse-to-fine stereopsis similar to the algorithm proposed by H.K. Nishihara (1984) and the resulting algorithm is applied to a number of stereo pairs.>

It has previously been demonstrated that detectors can be constructed that are sensitive to the 3D trajectory of a structure. The general approach is based upon a biological model for looming detectors proposed by K. Beverley and D. Regan (J. Physiol., vol.193, p.17-29, 1973). The computational model is based upon recent results in motion analysis and static stereopsis concerning the measurement of trajectory without prior form recognition. Some of the characteristics of these detectors are examined, and a number of experiments that show the responses of particular detectors to structure with different trajectories, disparities, and velocities are described. Some of the similarities and differences between the detectors presented and models for looming detectors present in biological vision systems are indicated.>