Modern cars monitor their surroundings and record video to deter intruders, but current surveillance systems operate independently without communication. Connected vehicles can share detected features of suspicious individuals, improving tracking and alerting approaching drivers before they park in vulnerable spots. This paper explores this use case, where connected vehicles exchange features over the network upon detecting suspicious individuals. Real-world data analysis shows that connected vehicle surveillance improves detection accuracy by 38%.

Recently, demand has been growing for development data in research on driver assistance systems. However, important scenes in research vary widely from person to person. For example, developers are interested in collision avoidance might be interested in pedestrian darting out or approaching vehicles. On the other hand, some developers may be interested in traffic scenes. Thus, user preferences vary and making the detection of important scenes are complex. We propose a novel approach to detect important scenes based on user's preferences, novelty of a driving scene and driving data. We annotate important scenes from the NuScenes dataset and confirmed improvement in accuracy from existing important scene detection model.

Fall-on-car incidents (e.g., trees or branches falling on a car) are an underestimated hazard during weather events, mainly affecting vehicles. Unfortunately, drivers are usually unaware of this danger until it occurs. On the other hand, connected vehicles can sense their surroundings, analyze this data with weather forecasts, and alert the driver if there is a risk of a fall-on-car incident. This paper focuses on this use case and demonstrates the Fall-on-Car Prevention (FoP) system. FoP system detects trees and tree branches and alerts drivers when windy conditions are forecasted, allowing early preventative action to be taken while parking. Our evaluation compared to 12 human experts demonstrates that the FoP system can enhance driver awareness of the risk of falling objects.

Interest in cooperative perception is growing quickly due to its remarkable performance in improving perception capabilities for connected and automated vehicles. This improvement is crucial, especially for automated driving scenarios in which perception performance is one of the main bottlenecks to the development of safety and efficiency. However, current cooperative perception methods typically assume that all collaborating vehicles have enough communication bandwidth to share all features with an identical spatial size, which is impractical for real-world scenarios. In this paper, we propose Adaptive Cooperative Perception, a new cooperative perception framework that is not limited by the aforementioned assumptions, aiming to enable cooperative perception under more realistic and challenging conditions. To support this, a novel feature encoder is proposed and named Pillar Attention Encoder. A pillar attention mechanism is designed to extract the feature data while considering its significance for the perception task. An adaptive feature filter is proposed to adjust the size of the feature data for sharing by considering the importance value of the feature. Experiments are conducted for cooperative object detection from multiple vehicle-based and infrastructure-based LiDAR sensors under various communication conditions. Results demonstrate that our method can successfully handle dynamic communication conditions and improve the mean Average Precision by 10.18% when compared with the state-of-the-art feature encoder.

Perceiving the environment is one of the most fundamental keys to enabling Cooperative Driving Automation, which is regarded as the revolutionary solution to addressing the safety, mobility, and sustainability issues of contemporary transportation systems. Although an unprecedented evolution is now happening in the area of computer vision for object perception, state-of-the-art perception methods are still struggling with sophisticated real-world traffic environments due to the inevitable physical occlusion and limited receptive field of single-vehicle systems. Based on multiple spatially separated perception nodes, Cooperative Perception (CP) is born to unlock the bottleneck of perception for driving automation. In this paper, we comprehensively review and analyze the research progress on CP, and we propose a unified CP framework. The architectures and taxonomy of CP systems based on different types of sensors are reviewed to show a high-level description of the workflow and different structures for CP systems. The node structure, sensing modality, and fusion schemes are reviewed and analyzed with detailed explanations for CP. A Hierarchical Cooperative Perception (HCP) framework is proposed, followed by a review of existing open-source tools that support CP development. The discussion highlights the current opportunities, open challenges, and anticipated future trends.

Modern cars can detect unsafe driving by comparing the observed behavior of the subject vehicle (i.e., rear vehicles) with normal driving. However, normal driving does not have a standard definition. It changes depending on the situation. In this work, we address this problem and propose edge-assisted unsafe driving detection. In our proposal, instead of learning normal driving, the edge infers the most common unsafe driving patterns. It then shares this knowledge with cars. Cars look for such patterns to detect unsafe driving. Analysis of real-world traffic data shows that edge-assisted unsafe driving detection could detect unsafe behavior of subject vehicles with 90% accuracy.

In this paper, a client-edge-cloud hierarchical federated learning (FL) model has been developed for connected vehicle applications. Generalized models are aggregated on the cloud server, while customized models trained on local data with similar data distribution are aggregated on the edge server, which mitigates the impact of data heterogeneity. To reduce the communication overhead of FL, clients will periodically update to the edge server and edge servers will periodically update to the cloud server. Moreover, we propose a mean field game-based probabilistic device selection scheme. Jointly considering their contributions and the population diversity, a fraction of devices will be selected to join the FL iteration. Taking driving range estimation as an example of connected vehicle applications in the experiment, we have shown that the proposed FL frameworks can increase the prediction accuracy by 28.9% with 5 times fewer clients’ participation, compared with the vanilla FL.

Unsafe driving has evolved into a public safety crisis. More than half of fatal crashes are due to distracted and aggressive driving. Modern cars have systems to monitor and notify drivers when erratic driving is detected. However, such systems do not help when other nearby vehicles drive unsafely. In this paper, we focus on this use case. We propose a nearby erratic driving detection method in which the ego vehicle observes its surroundings and identifies anomalous driving. We develop and test the proposed method through simulation. Then, we check the feasibility of the proposed method in field trials with multiple test vehicles. Evaluation results show that the proposed method can detect erratic driving on average 4 seconds before the risk of collision becomes maximum with 70% accuracy.

Modern cars can monitor rear vehicles and detect unsafe driving before the risk of rear-end collision becomes maximum. In this paper, we focus on this use case. We propose a Rear Vehicle Behavior Awareness (RVBA) system to prevent rear-end collisions. RVBA detects unsafe driving of rear vehicles and alerts the driver with guidance to reduce the risk of rearend collisions. We tested the RVBA in field experiments using a test vehicle. Experimental results show that RVBA can detect unsafe driving of rear vehicles in 4 seconds on average, with 70% accuracy, before the risk of rear-end collision becomes maximum. Index Terms–erratic movement patterns, rear-end collisions, rear vehicle behavior awareness, unsafe driving detection

Object perception plays a fundamental role in Cooperative Driving Automation (CDA) which is regarded as a revolutionary promoter for next-generation transportation systems. However, the vehicle-based perception may suffer from the limited sensing range and occlusion as well as low penetration rates in connectivity. In this paper, we propose Cyber Mobility Mirror (CMM), a next-generation real-world object perception system for 3D object detection, tracking, localization, and reconstruction, to explore the potential of roadside sensors for enabling CDA in the real world. The CMM system consists of six main components: i) the data pre-processor to retrieve and preprocess the raw data; ii) the roadside 3D object detector to generate 3D detection results; iii) the multi-object tracker to identify detected objects; iv) the global locator to generate geo-localization information; v) the mobile-edge-cloud-based communicator to transmit perception information to equipped vehicles, and vi) the onboard advisor to reconstruct and display the real-time traffic conditions. An automatic perception evaluation approach is proposed to support the assessment of data-driven models without human-labeling requirements and a CMM field-operational system is deployed at a real-world intersection to assess the performance of the CMM. Results from field tests demonstrate that our CMM prototype system can achieve 96.99% precision and 83.62% recall for detection and 73.55% ID-recall for tracking. High-fidelity real-time traffic conditions (at the object level) can be geo-localized with a root-mean-square error (RMSE) of$0.69m$and$0.33m$for lateral and longitudinal direction, respectively, and displayed on the GUI of the equipped vehicle with a frequency of$3-4 Hz$.

Aggressive driving is the leading cause of many fatal crashes. Ego vehicles should detect such dangerous driving behavior on other cars and guide drivers to mitigate collision risk. In this paper, we focus on that use case. We demonstrate a nearby aggressive driving detection system. In nearby aggressive driving detection, the ego vehicle observes the follower vehicle and detects aggressive driving behavior on the follower vehicle. It notifies its driver whenever the follower vehicle exhibits aggressive driving.

Multi-agent trajectory prediction is a challenging task because of the uncertainty of agents’ behaviors, interactions between agents, complex road geometry in urban environments, and imperfect/noisy agent histories. Although accurate prediction results are critical for safe and reliable intelligent driving applications (e.g., decision making, motion planning), some other applications may prefer light-weight and computation-efficient trajectory prediction models to handle dynamically changed environments. In this work, we propose a multi-agent, multi-modal Graph Attention Isomorphism Network (GAIN) based trajectory prediction framework to effectively understand and aggregate long-term interactions across agents. We also take the model complexity and computation efficiency into consideration. Experiments on both pedestrian and vehicle datasets demonstrated the effectiveness of our proposed method.

Aggressive driving became a public safety crisis in the USA. Aggressive drivers tailgate and weave among lanes, which may cause risky events ending up in collisions. Vehicles should be aware of such nearby aggressive driving and notify drivers to keep them away from aggressive ones. In this paper, we focus on that use case. The ego vehicle observes the movements of other nearby cars and detects aggressive driving. We tested the feasibility of the proposed approach through a simulation. Simulation results show that the ego vehicle could identify aggressive driving on other cars with about 94% accuracy.

Human–Robot Interaction challenges Artificial Intelligence in many regards: dynamic, partially unknown environments that were not originally designed for robots; a broad variety of situations with rich semantics to understand and interpret; physical interactions with humans that requires fine, low-latency yet socially acceptable control strategies; natural and multi-modal communication which mandates common-sense knowledge and the representation of possibly divergent mental models. This article is an attempt to characterise these challenges and to exhibit a set of key decisional issues that need to be addressed for a cognitive robot to successfully share space and tasks with a human. We identify first the needed individual and collaborative cognitive skills: geometric reasoning and situation assessment based on perspective-taking and affordance analysis; acquisition and representation of knowledge models for multiple agents (humans and robots, with their specificities); situated, natural and multi-modal dialogue; human-aware task planning; human–robot joint task achievement. The article discusses each of these abilities, presents working implementations, and shows how they combine in a coherent and original deliberative architecture for human–robot interaction. Supported by experimental results, we eventually show how explicit knowledge management, both symbolic and geometric, proves to be instrumental to richer and more natural human–robot interactions by pushing for pervasive, human-level semantics within the robot's deliberative system.

Robot path planning in human environments benefits significantly from considering more than obstacle avoidance, and recent works in this area proposed safety and comfort considerations. One shortcoming of current approaches is that humans' behavior is modeled as independent of robot's motions. In this work, we aim to give this anticipation ability to a robot by simulating people's reaction to robot's motion during planning. Our approach is based on extracting a static plan using A* search on the grid map by minimizing safety, disturbance and path length costs and then refining it by simulating humans' reaction using the Social Force Model. With two example scenarios in simulation and two on the real system, we provide qualitative examination of the resulting robot paths and demonstrate that robots can exhibit social behaviors that is not possible to model with standard approaches. This work serves as a primer for quantitative user studies, and we hope will urge future robot path planners to consider a richer set of social capabilities.

Comfortable object handover involves searching for a person, identifying them, and then actually handing them an object. It is a goal of many robotics applications, particularly in healthcare. But people can have a wide variety of physical, perceptual, or cognitive limitations. To address these variable patient conditions with an autonomous robot, a robot must adapt to these individual differences. In this work, we first interviewed nurses regarding common issues faced by healthcare providers. Then we designed a system that interfaces with an individual's electronic health record, and adapts its search and handover capabilities to improve the quality of object handover.

We present a novel approach for generating candidate handover positions for human receivers. The object handover problem involves several variables and is under-constrained. The sheer number of possibilities makes it nontrivial to find a good object handover position for existing algorithms. To address this problem, we introduce the use of a custom biomechanical model in a novel algorithm for generating a handover position. The biomechanical model serves as a tool for approximating the static strength a human receiver needs to exert in order to receive an object. A mobile manipulator can use the output of our algorithm when calculating a target position for handing over an object to a human. We show preliminary results involving electromyography data and joint moment values, as well as a proof-of-concept implementation with a mobile manipulator to fetch a toolbox to a receiver.

Guidance for human navigation using a vibro-tactile belt interface and robot-like motion planning

With recent advances in safe and compliant hardware and control, robots are close to finding their places in our homes. As the safety barrier between humans and robots is beginning to fade, the necessity to design pertinent robot behavior in human environments is becoming a crucial step. In order to obtain a safe, comfortable, and socially acceptable interaction, the robot should be engineered from top to bottom by considering the presence of the human. In this paper, we present a manipulation planning framework and its implementation human-aware manipulation planner. This planner generates paths not only safe but comfortable and “socially acceptable” as well by reasoning explicitly on human's kinematics, vision field, posture, and preferences. The planner, which is applied into “robot handing over an object” scenarios, breaks the human centric interaction that depends mostly on human effort and allows the robot to take initiative by computing automatically where the interaction takes place, thus decreasing the cognitive weight of interaction on human side.

This paper addresses the motion planning problem while considering Human-Robot Interaction (HRI) constraints. The proposed planner generates collision-free paths that are acceptable and legible to the human. The method extends our previous work on human-aware path planning to cluttered environments. A randomized cost-based exploration method provides an initial path that is relevant with respect to HRI and workspace constraints. The quality of the path is further improved with a local path-optimization method. Simulation results on mobile manipulators in the presence of humans demonstrate the overall efficacy of the approach.

If robots are to cooperate with humans in an increasingly human-like manner, then significant progress must be made in their abilities to observe and learn to perform novel goal directed actions in a flexible and adaptive manner. The current research addresses this challenge. In CHRIS.I [1], we developed a platform-independent perceptual system that learns from observation to recognize human actions in a way which abstracted from the specifics of the robotic platform, learning actions including “put X on Y” and “take X”. In the current research, we extend this system from action perception to execution, consistent with current developmental research in human understanding of goal directed action and teleological reasoning. We demonstrate the platform independence with experiments on three different robots. In Experiments 1 and 2 we complete our previous study of perception of actions “put” and “take” demonstrating how the system learns to execute these same actions, along with new related actions “cover” and “uncover” based on the composition of action primitives “grasp X” and “release X at Y”. Significantly, these compositional action execution specifications learned on one iCub robot are then executed on another, based on the abstraction layer of motor primitives. Experiment 3 further validates the platform-independence of the system, as a new action that is learned on the iCub in Lyon is then executed on the Jido robot in Toulouse. In Experiment 4 we extended the definition of action perception to include the notion of agency, again inspired by developmental studies of agency attribution, exploiting the Kinect motion capture system for tracking human motion. Finally in Experiment 5 we demonstrate how the combined representation of action in terms of perception and execution provides the basis for imitation. This provides the basis for an open ended cooperation capability where new actions can be learned and integrated into shared plans for cooperation. Part of the novelty of this research is the robots' use of spoken language understanding and visual perception to generate action representations in a platform independent manner based on physical state changes. This provides a flexible capability for goal-directed action imitation.

In daily human interactions spatial reasoning occupies an important place. With this ability we can build relations between objects and people, and we can predict the capabilities and the knowledge of the people around us. An interactive robot is also expected to have these abilities in order to establish an efficient and natural interaction. In this paper we present a situation assessment reasoner, based on spatial reasoning and perspective taking, which generates on-line relations between objects and agents in the environment. Being fully integrated to a complete architecture, this reasoner sends the generated symbolic knowledge to a fact data base which is built on the basis on an ontology and which is accessible to the entire system. This work is also part of a broader effort to develop a complete decisional framework for human-robot interactive task achievement.

Humans constantly generate and solve ambiguities while interacting with each other in their every day activities. Hence, having a robot that is able to solve ambiguous situations is essential if we aim at achieving a fluent and acceptable human-robot interaction. We propose a strategy that combines three mechanisms to clarify ambiguous situations generated by the human partner. We implemented our approach and successfully performed validation tests in several different situations both, in simulation and with the HRP-2 robot.

In human-robot interaction, a robot must be prepared to handle possible ambiguities generated by a human partner. In this work we propose a set of strategies that allow a robot to identify the referent when the human partner refers to an object giving incomplete information, i.e. an ambiguous description. Moreover, we propose the use of an ontology to store and reason on the robot's knowledge to ease clarification, and therefore, improve interaction. We validate our work through both simulation and two real robotic platforms performing two tasks: a daily-life situation and a game.

Robot path planning has traditionally concentrated on collision-free paths. For robots that collaborate closely with humans, however, the situation is different in two respects: 1) the humans in the robot's environment are not randomly moving objects, but cognitive beings who can deliberately make way for a robot to pass and 2) the quality of a navigation plan depends less on quantitative efficiency criteria, but rather on the acceptance of humans. In this paper, we introduce a robot navigation approach that takes into account human-centered requirements and the collaborative nature of the interaction between the human and the robot.

Human Robot collaborative task achievement requires adapted tools and algorithms for both decision making and motion computation. The human presence as well as its behavior must be considered and actively monitored at the decisional level for the robot to produce synchronized and adapted behavior. Additionally, having a human within the robot range of action introduces security constraints as well as comfort considerations which must be taken into account at the motion planning and control level. This paper presents a robotic architecture adapted to human robot interaction and focuses on two tools: a human aware manipulation planner and a supervision system dedicated to collaborative task achievement.

Robots' interaction with humans raises new issues for geometrical reasoning where the humans must be taken explicitly into account. We claim that a human-aware motion system must not only elaborate safe robot motions, but also synthesize good, socially acceptable and legible movement. This paper focuses on a manipulation planner and a placement mechanism that take explicitly into account its human partners by reasoning about their accessibility, their vision field and their preferences. This planner is part of a human-aware motion and manipulation planning and control system that we aim to develop in order to achieve motion and manipulation tasks in presence or in synergy with humans.

Robot navigation in the presence of humans raises new issues for motion planning and control when the humans must be taken explicitly into account. We claim that a human aware motion planner (HAMP) must not only provide safe robot paths, but also synthesize good, socially acceptable and legible paths. This paper focuses on a motion planner that takes explicitly into account its human partners by reasoning about their accessibility, their vision field and their preferences in terms of relative human-robot placement and motions in realistic environments. This planner is part of a human-aware motion and manipulation planning and control system that we aim to develop in order to achieve motion and manipulation tasks in the presence or in synergy with humans.

This paper presents the combined results of two studies that investigated how a robot should best approach and place itself relative to a seated human subject. Two live Human Robot Interaction (HRI) trials were performed involving a robot fetching an object that the human had requested, using different approach directions. Results of the trials indicated that most subjects disliked a frontal approach, except for a small minority of females, and most subjects preferred to be approached from either the left or right side, with a small overall preference for a right approach by the robot. Handedness and occupation were not related to these preferences. We discuss the results of the user studies in the context of developing a path planning system for a mobile robot.

The presence of humans should be explicitly taken into account in all steps of robot's design and particularly for robot motion. The robot should reason about human partner's accessibility, his vision field and potential shared motions and behave as a social being by respecting social rules and protocols. This paper describes the algorithms and results of a navigation planner that takes into account the human presence explicitly. This planner is part of a human-aware motion and manipulation planning and control system that we aim to develop in order to achieve motion and manipulation tasks in presence and/or in synergy with human

The presence of humans in the robot environment brings new challenges to the robotic research. From low level functions to high level planners, clearly the human has to be taken into account in all the layers of the robot control system. Indeed, the robot has to behave socially in order to interact friendly with its human partners. This paper describes the development of several components (human detection and tracking, planning and supervision) that take into account humans explicitly with some preliminary results of their integration.