In this paper, we study the problem of gesture recognition as a method for divers to communicate with an underwater robot. Gesture is a common method of communication between divers, and yet autonomous underwater vehicles have very limited capacity to understand gesture given lighting and visibility constraints (e.g., from water turbidity and diver depth). Traditional deep learning methods are limited in this domain because of a lack of sufficient training data. We show that it is not enough to learn a gesture in a laboratory setting, because the appearance changes dramatically underwater. We show how hyperdimensional computing can solve this problem by permitting hypervectors to serve as abstract representations of gestures, yielding rapid adaptation to new environments and new gestures. We experimentally verify this approach using a novel dataset of 6 diving relevant gestures. We show that we can accurately adapt to a gesture learned in a laboratory setting to work with a gesture observed underwater. Our approach compares favorably to a ResNet-18, which performs well in laboratory conditions (91.9% accuracy), but performs poorly underwater (53.9% accuracy). Our proposed approach is capable of rapid adaptation, resulting in an accuracy of 83.8% on underwater gestures with just one additional example from each class added to the support set. Finally, we also show the ability to adapt to new gestures not present in our original training set. We use hypervectors to learn new gestures from the Sign Language MNIST dataset, providing a high level of accuracy with a limited amount of training data.

Dorsal hand vein recognition, and vascular bio-metrics in general, is becoming an increasingly reliable and trusted way to recognize people. However, previous technology limits vascular biometrics to cooperative individuals - requiring a person to hold their hand still next to a near infrared sensor [16]. In this paper, we show that we can extend the range and utility of this biometric using long wave infrared (LWIR) imaging. LWIR shows the thermal gradient on the dorsal side of the hand revealing the patterns related to the underlying vascular structure. To study this problem, we introduce Thermal CoAct, a database of people performing common actions while their hands are sensed in the LWIR region using two different sensing technologies. To identify a person, we fine-tune a Vision Transformer (ViT). Our experimental results show that we can successfully recognize unconstrained hands in both open-world and closed-world scenarios. We explore how different sensing technologies as well as different styles of enrollment affect performance. When paired with a good enrollment strategy and sensor, we have achieved a rank-1 accuracy of 96.7% for closed-world scenarios and equal error rates (EER) of 3.1 % for open-world scenarios.

This paper demonstrates the effectiveness of learning based models for accurate, and reliable robot to human handoffs in various HRI scenarios. Specifically we bench marked a neuromorphic spiking neural network and a time series k-nearest neighbors classifier against traditional hand crafted force threshold methods. These models use linear force in the x, y, and z direction, as well as torque about the x, y, and z axis at the end effector of the robot arm to make handoff predictions. This paper demonstrates that these learning based methods are more robust to noise which occurs during operational use. We applied our algorithms to both stationary handoffs (stationary robot) and moving handoffs (robot walking). We believe that our evaluation is the first to examine walking handoffs. We evaluated all models in tests which determined the accuracy, precision, recall, f1, and average execution time for handoff events, noise events, and no event tests. We find that the SLAYER spiking neural network model performed the best across both walking and stationary handoffs for the majority of the evaluation criteria. Our results suggest that neuromorphic spiking neural networks are strong contenders for applications in time series, event based HRI applications.

Learning to recognize new objects in real time in unconstrained environments presents significant challenges for robotic platforms. We present a meta-learning solution to this problem as well as a registered image and events dataset to facilitate work in this domain. Our solution uses interactive motion to isolate the object, and motion-based saliency (from events) to select relevant keypoints from a high-resolution RGB image. Salient keypoints are then passed to a meta-learner to classify the object type. We show that using our interactive isolation and keypoint selection approach, we outperform existing techniques by 6-20%.

Research into person attributes recognition has focused on approaches to describe a person in terms of their appearance. Typically, this includes a wide range of traits including age, gender, clothing, and footwear. Although this could be used in a wide variety of scenarios, it generally is applied to video surveillance, where attribute recognition is impacted by low resolution, and other issues such as variable pose, occlusion and shadow. Recent approaches have used deep convolutional neural networks (CNNs) to improve the accuracy in person attribute recognition. However, many of these networks are relatively shallow and it is unclear to what extent they use contextual cues to improve classification accuracy. This paper builds upon prior research by proposing to use a modified ResNet architecture with calibrations that permit us to train networks that are deeper than previously published approaches. Interpretation suggests that this deeper architectures allows the network to take more contextual information into consideration, which helps to improve classification accuracy and generalizability. We present experimental analysis and results for whole body attributes using the PA-100K and PETA datasets and facial attributes using the CelebA dataset.

In surveillance images, soft biometric attributes have been demonstrated to be quite effective in the problem of person re-identification. Many of these attributes can vary greatly, which has motivated a number of hand crafted features designed to recognize individual attributes. Although deep learning is generally useful in learning features appropriate for classification, it usually requires more data to train than what is available in most person re-identification databases. In this paper, we propose a residual network (MAResNet) deeper than current pedestrian attribute recognition, which we use to recognize multiple attributes simultaneously. The proposed network is both efficient and accurate. We recognize attributes at a rate of 271 FPS while simultaneously outperforming state of the art in attribute recognition on the PETA dataset.

The ability to perform robust, precise, real-time visual recognition is extremely critical for the use of robotic systems in real-world applications. This paper explores the use of Convolution Neural Networks (CNN) and human assisted training in teaching a robot to recognize novel objects. We investigated the impact of providing instructions to a human teacher during a training scenario for novel objects. Participants in the naïve condition were provided verbal instructions by the robot, and participants in the embodied condition were provided embodied demonstrations by the robot. The results showed that a vision system trained by participants with embodied instructions clearly outperformed a system trained by naïve participants. The latest computer vision techniques combined with human assisted teaching was found to provide excellent results for novel object recognition.

Many recent works have successfully leveraged motion information (i.e., dense optical flow) for a variety of problems. In this paper, we introduce a methodology to capture motion information using high-speed event-based cameras combined with convolutional neural networks (CNN). Our motion event features (MEFs) succinctly capture motion magnitude and direction in a form suitable for input into a CNN. We demonstrate the broad applicability of MEFs across two disparate problems: action recognition, and autonomous robot reactive control.

In Navy human firefighting teams, touch is used extensively to communicate among teammates. In noisy, chaotic, and visually challenging environments, such as among fires on Navy ships, this is the only reliable means of communication. The overarching goal of this work is to augment Navy firefighting teams with an autonomous robot serving as a nozzle operator; to accomplish this, the robot must understand the tactile gestures of its human teammates. Preliminary results recognizing touch gestures have indicated the potential of such an autonomous system to serve as a nozzle operator in human-centric firefighting scenarios.

Knowing the terrain is vital for small autonomous robots traversing unstructured outdoor environments. We present a technique using 3D laser point clouds combined with RGB camera images to classify terrain into four pre-defined classes: grass, sand, concrete, and metal. Our technique first segments the point cloud into distinct regions and then applies a simple classifier to determine the classification of each region. We demonstrate three classification and four segmentation algorithms on five outdoor environments. Classification and segmentation algorithms which use more information outperform information poor combinations.

This video submission demonstrates cooperative human-robot firefighting. A human team leader guides the robot to the fire using a combination of speech and gesture.

Human robot interaction presents a unique set of challenges for biometric person identification. During normal interactions between the robot and a user, a tremendous amount of information is available for identification. Our objective is to use this information to identify users quickly and accurately during interactions with a robot. We present our approach for multimodal person identification using Markov logic networks (MLN). We use appearance, clothing, speaker recognition, and face recognition to identify a person during an interaction where they are speaking to the robot. We demonstrate the effectiveness of our approach using sequences of individuals speaking freely on a topic of their choosing.

Person identification is the problem of identifying an individual that a computer system is seeing, hearing, etc. Typically this is accomplished using models of the individual. Over time, however, people change. Unless the models stored by the robot change with them, those models will became less and less reliable over time. This work explores automatic updating of person identification models in the domain of speaker recognition. By fusing together tracking and recognition systems from both visual and auditory perceptual modalities, the robot can robustly identify people during continuous interactions and update its models in real-time, improving rates of speaker classification.

The interest in face recognition is moving toward real-world applications and uncontrolled sensing environments. An important application of interest is automated surveillance, where the objective is to recognize and track people who are on a watchlist. For this open world application, a large number of cameras that are increasingly being installed at many locations in shopping malls, metro systems, airports, etc., will be utilized. While a very large number of people will approach or pass by these surveillance cameras, only a small set of individuals must be recognized. That is, the system must reject every subject unless the subject happens to be on the watchlist. While humans routinely reject previously unseen faces as strangers, rejection of previously unseen faces has remained a difficult aspect of automated face recognition. In this paper, we propose an approach motivated by human perceptual ability of face recognition which can handle previously unseen faces. Our approach is based on identifying the decision region(s) in the face space which belong to the target person(s). This is done by generating two large sets of borderline images, projecting just inside and outside of the decision region. For each person on the watchlist, a dedicated classifier is trained. Results of extensive experiments support the effectiveness of our approach. In addition to extensive experiments using our algorithm and prerecorded images, we have conducted considerable live system experiments with people in realistic environments.

We propose a novel approach to gait analysis using the independent components of motion. Our motion representation uses the amount of translation in small image patches. For each subject in the training set, several short image sequences are selected at random. Spatiotemporal independent component analysis (stICA) estimates independent components (ICs) of the training sequences. Given an unknown subject, we compute stIC coefficients for all image subsequences. These coefficients are compared with the training set using the cosine similarity measure. We demonstrate feasibility by using the nearest neighbor to classify the gender of a subject.