While the design of autonomous vehicle systems often centers on efficiency and reliability, user experience and trust play a critical role in adoption and long-term use. This paper presents a foundational step in that direction by introducing an Interactive Reinforcement Learning algorithm, termed Trust-Aware Reward Shaping, or TARS in short, to incorporate trust dynamics into vehicle automation design. The TARS algorithm leverages real-time trust as an additional reward signal during training of the automated trajectory generation system, with lane change as the use case. It, therefore, enables customization based on the user-preferred driving style (aggressive, neutral, or conservative) and trust dynamics (vigilant, confident, or skeptical). Consequently, the algorithm successfully generates emergent behavior that dynamically responds to trust changes, such as a more aggressive style when users show over-trust tendency, and a more neutral style when users demonstrate under-trust or skeptical behaviors. It also predicts real-world driving behaviors observed in naturalistic driving data, and outperforms non-customized, expert-coded, and game-play trust-aware RL methods, as well as baseline inverse RL methods, especially for aggressive and neutral driving styles.

We present a new framework to measure the intrinsic properties of (deep) neural networks. While we focus on convolutional networks, our framework can be extrapolated to any network architecture. In particular, we evaluate two network properties, namely, capacity, which is related to expressivity, and compression, which is related to learnability. Both these properties depend only on the network structure and are independent of the network parameters. To this end, we propose two metrics: the first one, called layer complexity, captures the architectural complexity of any network layer; and, the second one, called layer intrinsic power, encodes how data are compressed along the network. The metrics are based on the concept of layer algebra, which is also introduced in this article. This concept is based on the idea that the global properties depend on the network topology, and the leaf nodes of any neural network can be approximated using local transfer functions, thereby allowing a simple computation of the global metrics. We show that our global complexity metric can be calculated and represented more conveniently than the widely used Vapnik-Chervonenkis (VC) dimension. We also compare the properties of various state-of-the-art architectures using our metrics and use the properties to analyze their accuracy on benchmark image classification datasets.

Recognition of occluded objects in unseen and unstructured indoor environments is a challenging problem for mobile robots. To address this challenge, we propose a new descriptor, Topological features Of Point cloud Slices (TOPS), for point clouds generated from depth images and an accompanying recognition framework, TOPS for Human-inspired Object Recognition (THOR), inspired by human reasoning. The descriptor employs a novel slicing-based approach to compute topological features from filtrations of simplicial complexes using persistent homology and facilitates reasoning-based recognition using object unity. Apart from a benchmark dataset, we report performance on a new dataset, the UW Indoor Scenes (UW-IS) Occluded dataset, curated using commodity hardware to reflect real-world scenarios with different environmental conditions and degrees of object occlusion. THOR outperforms state-of-the-art methods on both the datasets and achieves substantially higher recognition accuracy for all the scenarios of the UW-IS Occluded dataset. Therefore, THOR is a promising step toward robust recognition in low-cost robots, meant for everyday use in indoor settings.

Bird's Eye View (BEV) representations are tremendously useful for perception-related automated driving tasks. However, generating BEVs from surround-view fisheye camera images is challenging due to the strong distortions introduced by such wide-angle lenses. We take the first step in addressing this challenge and introduce a baseline, F2BEV, to generate discretized BEV height maps and BEV semantic segmentation maps from fisheye images. F2BEV consists of a distortion-aware spatial cross attention module for querying and consolidating spatial information from fisheye image features in a transformer-style architecture followed by a task-specific head. We evaluate single-task and multi-task variants of F2BEV on our synthetic FB-SSEM dataset, all of which generate better BEV height and segmentation maps (in terms of the IoU) than a state-of-the-art BEV generation method operating on undistorted fisheye images. We also demonstrate discretized height map generation from real-world fisheye images using F2BEV. Our dataset is publicly available at https://github.com/volvo-cars/FB-SSEM-dataset

The goal of this study is to provide a framework, using hidden semi-Markov models, for modeling a driver’s response time after an alert is provided in manual driving. Given the plethora of alerts and warning within a vehicle, there is a need to understand when a driver will respond after an alert is provided. Data from a previous driving simulator study, where drivers were interacting with an in-vehicle information system (IVIS) were used for model training. The final data set included 16 participants, with 288 task initiations. The proposed model could predict a driver’s response time accurately using only a small portion of the available data, and had a mean absolute error of 0.51 seconds with 84% of predictions within an absolute error of 1 second. This framework has applicability in mitigating the risk of transitions in driver distraction. This includes transitions from the road to the secondary task and back to the road.

We propose a new approach to Human Activity Evaluation (HAE) in long videos using graph-based multi-task modeling. Previous works in activity evaluation either directly compute a metric using a detected skeleton or use the scene information to regress the activity score. These approaches are insufficient for accurate activity assessment since they only compute an average score over a clip, and do not consider the correlation between the joints and body dynamics. Moreover, they are highly scene-dependent which makes the generalizability of these methods questionable. We propose a novel multi-task framework for HAE that utilizes a Graph Convolutional Network backbone to embed the interconnections between human joints in the features. In this framework, we solve the Human Activity Segmentation (HAS) problem as an auxiliary task to improve activity assessment. The HAS head is powered by an Encoder-Decoder Temporal Convolutional Network to semantically segment long videos into distinct activity classes, whereas, HAE uses a Long-Short-Term-Memory-based architecture. We evaluate our method on the UW-IOM and TUM Kitchen datasets and discuss the success and failure cases in these two datasets.

Sensitivity analysis (SA) is an important aspect of process automation. It often aims to identify the process inputs that influence the process output’s variance significantly. Existing SA approaches typically consider the input–output relationship as a black box and conduct extensive random sampling from the actual process or its high-fidelity simulation model to identify the influential inputs. In this article, an alternate, novel approach is proposed using a sparse polynomial chaos expansion-based model for a class of input–output relationships represented as directed acyclic networks. The model exploits the relationship structure by recursively relating a network node to its direct predecessors to trace the output variance back to the inputs. It, thereby, estimates the Sobol indices that measure the influence of each input on the output variance, accurately and efficiently. Theoretical analysis establishes the validity of the model as the prediction of the network output converges in probability to the true output under certain regularity conditions. Empirical evaluation of two manufacturing processes and a flooding process shows that the model estimates the Sobol indices accurately with far fewer observations than state-of-the-art black-box methods.Note to Practitioners—This article is motivated by the problem of automated identification of the inputs that influence the variance of the output for networked processes without feedback control. Such processes arise in various natural and engineered systems, of which manufacturing operations and flood mitigation are of particular interest to us, where the output variance represents the uncertainty in productivity, quality, or cost. Therefore, influential inputs’ identification allows us to quantify the effects of the various process parameters, such as operating conditions and physical properties, in determining the uncertainties in the process outputs. We show that our identification method is guaranteed to quantify the effects accurately and is expected to do so more efficiently (with fewer experimental observations) than widely used stochastic sampling techniques. In the future, we will like to evaluate the usefulness of the developed method on large-scale manufacturing and supply chain networks in actual production facilities as well as on critical infrastructures subject to cascading failures.

Topological data analysis (TDA) has emerged as one of the most promising techniques to reconstruct the unknown shapes of high-dimensional spaces from observed data samples. TDA, thus, yields key shape descriptors in the form of persistent topological features that can be used for any supervised or unsupervised learning task, including multi-way classification. Sparse sampling, on the other hand, provides a highly efficient technique to reconstruct signals in the spatial-temporal domain from just a few carefully-chosen samples. Here, we present a new method, referred to as the Sparse-TDA algorithm, that combines favorable aspects of the two techniques. This combination is realized by selecting an optimal set of sparse pixel samples from the persistent features generated by a vector-based TDA algorithm. These sparse samples are selected from a low-rank matrix representation of persistent features using QR pivoting. We show that the Sparse-TDA method demonstrates promising performance on three benchmark problems related to human posture recognition and image texture classification.

Planning operations across a number of domains can be considered as resource allocation problems with timing constraints. An unexplored instance of such a problem domain is the aircraft carrier flight deck, where, in current operations, replanning is done without the aid of any computerized decision support. Rather, veteran operators employ a set of experience-based heuristics to quickly generate new operating schedules. These expert user heuristics are neither codified nor evaluated by the United States Navy; they have grown solely from the convergent experiences of supervisory staff. As unmanned aerial vehicles (UAVs) are introduced in the aircraft carrier domain, these heuristics may require alterations due to differing capabilities. The inclusion of UAVs also allows for new opportunities for on-line planning and control, providing an alternative to the current heuristic-based replanning methodology. To investigate these issues formally, we have developed a decision support system for flight deck operations that utilizes a conventional integer linear program-based planning algorithm. In this system, a human operator sets both the goals and constraints for the algorithm, which then returns a proposed schedule for operator approval. As a part of validating this system, the performance of this collaborative human-automation planner was compared with that of the expert user heuristics over a set of test scenarios. The resulting analysis shows that human heuristics often outperform the plans produced by an optimization algorithm, but are also often more conservative.

Manipulation of microscopic objects, especially biological objects and microelectromechanical systems (MEMS) components, has become an important area of robotics research over the past several years. Automation is necessary as it is challenging to manually control the microobjects due to the scaling effect of the surface forces, stochastic motion of objects in fluid media, and uncertainty associated with object state estimation. Automation requires real-time control of the position, orientation, and force applied by each of the operational manipulators, as governed by the system-level objectives of optimizing resource, time, and effort, by planning suitable actions for the manipulated objects. In this paper, we provide a survey of the research in planning and control of such automated micromanipulation operations. We present a broad taxonomy based on the underlying approach, and discuss the salient features and experimental success of each research effort. We also identify the major limitations and common trends across all the approaches, discuss the effectiveness of an approach depending on the operation characteristics, and outline promising future research directions.

Automated transport of multiple particles using optical tweezers requires real-time path planning to move them in coordination by avoiding collisions among themselves and with randomly moving obstacles. This paper develops a decoupled and prioritized path planning approach by sequentially applying a partially observable Markov decision process algorithm on every particle that needs to be transported. We use an iterative version of a maximum bipartite graph matching algorithm to assign given goal locations to such particles. We then employ a three-step method consisting of clustering, classification, and branch and bound optimization to determine the final collision-free paths. We demonstrate the effectiveness of the developed approach via experiments using silica beads in a holographic tweezers setup. We also discuss the applicability of our approach and challenges in manipulating biological cells indirectly by using the transported particles as grippers.

This paper describes a new model for understanding natural language commands given to autonomous systems that perform navigation and mobile manipulation in semi-structured environments. Previous approaches have used models with fixed structure to infer the likelihood of a sequence of actions given the environment and the command. In contrast, our framework, called Generalized Grounding Graphs, dynamically instantiates a probabilistic graphical model for a particular natural language command according to the command's hierarchical and compositional semantic structure. Our system performs inference in the model to successfully find and execute plans corresponding to natural language commands such as "Put the tire pallet on the truck." The model is trained using a corpus of commands collected using crowdsourcing. We pair each command with robot actions and use the corpus to learn the parameters of the model. We evaluate the robot's performance by inferring plans from natural language commands, executing each plan in a realistic robot simulator, and asking users to evaluate the system's performance. We demonstrate that our system can successfully follow many natural language commands from the corpus.

Automated particle transport using optical tweezers requires the use of motion planning to move the particle while avoiding collisions with randomly moving obstacles. This paper describes a stochastic dynamic programming based motion planning framework developed by modifying the discrete version of an infinite-horizon partially observable Markov decision process algorithm. Sample trajectories generated by this algorithm are presented to highlight effectiveness in crowded scenes and flexibility. The algorithm is tested using silica beads in a holographic tweezer set-up and data obtained from the physical experiments are reported to validate various aspects of the planning simulation framework. This framework is then used to evaluate the performance of the algorithm under a variety of operating conditions.