UAV-based wireless networks can be deployed to provide a network coverage to users who have no or poor network connection. Unlike traditional model-based approaches that require predefined assumptions before UAV deployment, reinforcement learning (RL) offers a promising alternative but requires a realistic simulator for training the proposed strategies. None of the existing open-source simulators provide both realistic wireless communications while enabling the training, in a cluttered environment, of multi-UAVs movement strategies using RL. Thus, in this paper, we present a simulator where we focus on improving the modeling of the network access part, i.e., the communications between the UAVs and the users, by integrating signal propagation, physical rate adaptation and medium access sharing models. We evaluate the performance of a standard independent RL algorithm trained in our simulator across various use case scenarios, and compare the results obtained using different learning objectives. Results show that the classical formulation, commonly found in the literature and based on a a simplified wireless network model, underperforms in terms of communication quality.

Successfully addressing a wide variety of tasks is a core ability of autonomous agents, requiring flexibly adapting the underlying decision-making strategies and, as we ar-gue in this work, also adapting the perception modules. An analogical argument would be the human visual system, which uses top-down signals to focus attention determined by the current task. Similarly, we adapt pretrained large vision models conditioned on specific downstream tasks in the context of multitask policy learning. We introduce task-conditioned adapters that do not require finetuning any pretrained weights, combined with a single policy trained with behavior cloning and capable of addressing multiple tasks. We condition the visual adapters on task embeddings, which can be selected at inference if the task is known, or alternatively inferred from a set of example demonstrations. To this end, we propose a new optimization-based estimator. We evaluate the method on a wide variety of tasks from the CortexBench benchmark and show that, compared to existing work, it can be addressed with a single policy. In particular, we demonstrate that adapting visual features is a key design choice and that the method generalizes to unseen tasks given a few demonstrations.

Implicit representations such as Neural Radiance Fields (NeRF) allow to map color, density and semantics in a 3D scene through a continuous neural function. However, these models typically require manual and careful human data collection for training. This paper addresses the problem of active exploration for autonomous NeRF construction. We study how an agent can learn to efficiently explore an unknown 3D environment so that the data collected during autonomous exploration enables the learning of a high-quality neural implicit map representation. The quality of the learned representation is evaluated on four robotics-related downstream tasks: classical viewpoint rendering, map reconstruction, planning, and pose refinement. We compare the impact of different exploration strategies including frontier-based and learning-based approaches (end-to-end and modular) with different reward functions tailored to this problem. Empirical results show that NeRFs can be trained on actively collected data using just a single episode of experience in an unseen environment and that AutoNeRF, a modular exploration policy trained with reinforcement learning, enables obtaining a higher-quality NeRF for the considered downstream robotic tasks. Finally, we show that with AutoNeRF an agent can be deployed to a previously unknown scene and then automatically improve its navigation performance by adapting to the scene through a cycle of exploration, reconstruction, and policy finetuning.

Understanding and mapping a new environment are core abilities of any autonomously navigating agent. While classical robotics usually estimates maps in a stand-alone manner with SLAM variants, which maintain a topological or metric representation, end-to-end learning of navigation keeps some form of memory in a neural network. Networks are typically imbued with inductive biases, which can range from vectorial representations to birds-eye metric tensors or topological structures. In this work, we propose to structure neural networks with two neural implicit representations, which are learned dynamically during each episode and map the content of the scene: (i) the Semantic Finder predicts the position of a previously seen queried object; (ii) the Occupancy and Exploration Implicit Representation encapsulates information about explored area and obstacles, and is queried with a novel global read mechanism which directly maps from function space to a usable embedding space. Both representations are leveraged by an agent trained with Reinforcement Learning (RL) and learned online during each episode. We evaluate the agent on Multi-Object Navigation and show the high impact of using neural implicit representations as a memory source.

Artificial Intelligence systems have a significant impact on human lives. Machine Ethics tries to align these systems with human values, by integrating “ethical considerations”. However, most approaches consider a single objective, and thus cannot accommodate different, contextual human preferences. Multi-Objective Reinforcement Learning algorithms account for various preferences, but they often are not intelligible nor contextual (e.g., weighted preferences). Our novel approach identifies dilemmas, presents them to users, and learns to settle them, based on intelligible and contextualized preferences over actions. We intend to maximize understandability and opportunities for user-system co-construction by showing dilemmas, and triggering interactions, thus empowering users. The block-based architecture enables leveraging simple mechanisms that can be updated and improved. Validation on a Smart Grid use-case shows that our algorithm finds actions for various trade-offs, and quickly learns to settle dilemmas, reducing the cognitive load on users.

In the context of visual navigation, the capacity to map a novel environment is necessary for an agent to exploit its observation history in the considered place and efficiently reach known goals. This ability can be associated with spatial rea-soning, where an agent is able to perceive spatial relationships and regularities, and discover object characteristics. Recent work introduces learnable policies parametrized by deep neural networks and trained with Reinforcement Learning (RL). In classical RL setups, the capacity to map and reason spatially is learned end-to-end, from reward alone. In this setting, we introduce supplementary supervision in the form of auxiliary tasks designed to favor the emergence of spatial perception capabilities in agents trained for a goal-reaching downstream objective. We show that learning to estimate metrics quantifying the spatial relationships between an agent at a given location and a goal to reach has a high positive impact in Multi-Object Navigation settings. Our method significantly improves the performance of different baseline agents, that either build an explicit or implicit representation of the environment, even matching the performance of incomparable oracle agents taking ground-truth maps as input. A learning-based agent from the literature trained with the proposed auxiliary losses was the winning entry to the Multi-Object Navigation Challenge, part of the CVPR 2021 Embodied AI Workshop.

Routing delivery vehicles to serve customers in dynamic and uncertain environments like dense city centers is a challenging task that requires robustness and flexibility. Most existing approaches to routing problems produce solutions offline in the form of plans, which only apply to the situation they have been optimized for. Instead, we propose to learn a policy that provides decision rules to build the routes from online measurements of the environment state, including the customers configuration itself. Doing so, we can generalize from past experiences and quickly provide decision rules for new instances of the problem without re-optimizing any parameters of our policy. The difficulty with this approach comes from the complexity to represent this state. In this paper, we introduce a sequential multi-agent decision-making model to formalize the description and the temporal evolution of a Dynamic and Stochastic Vehicle Routing Problem. We propose a variation of Deep Neural Network using Attention Mechanisms to learn generalizable representation of the state and output online decision rules adapted to dynamic and stochastic information. Using artificially-generated data, we show promising results in these dynamic and stochastic environments, while staying competitive in deterministic ones compared to offline classical heuristics.

International audience

This paper aims to use a fleet of mobile robots, each embedding a camera, to optimize the observation of a human dynamic scene. The scene is defined as a sequence of activities, performed by a person in a same place. Mobile robots have to cooperate to find a spatial configuration around the scene that maximizes the joint observation of the human pose skeleton. It is assumed that the robots can communicate but have no map of the environment and no external localisation. This paper presents a concentric navigation topology allowing to keep easily each robot camera towards the scene. This topology is combined with an incremental mapping of the environment in order to limit the complexity of the exploration state space. We also introduce the marginal contribution of each robot observation, to facilitate stability in the search, while the exploration is guided by a meta-heuristics. We developped a simulator that uses skeleton data from real human pose captures. It allows to compare the variants of the approach and to show its features such as adaptation to the dynamic of the scene and robustness to the noise in the observations.

Recent works on multi-agent sequential decision making using decentralized partially observable Markov decision processes have been concerned with interaction-oriented resolution techniques and provide promising results. These techniques take advantage of local interactions and coordination. In this paper, we propose an approach based on an interaction-oriented resolution of decentralized decision makers. To this end, distributed value functions (DVF) have been used by decoupling the multi-agent problem into a set of individual agent problems. However existing DVF techniques assume permanent and free communication between the agents. In this paper, we extend the DVF methodology to address full local observability, limited share of information and communication breaks. We apply our new DVF in a real-world application consisting of multi-robot exploration where each robot computes locally a strategy that minimizes the interactions between the robots and maximizes the space coverage of the team even under communication constraints. Our technique has been implemented and evaluated in simulation and in real-world scenarios during a robotic challenge for the exploration and mapping of an unknown environment. Experimental results from real-world scenarios and from the challenge are given where our system was vice-champion.

This paper addresses the problem of exploring an unknown area with a team of autonomous robots using decentralized decision making techniques. The localization aspect is not considered and it is assumed the robots share their positions and have access to a map updated with all explored areas. A key problem is then the coordination of decentralized decision processes: each individual robot must choose appropriate exploration goals so that the team simultaneously explores different locations of the environment. We formalize this problem as a Decentralized Markov Decision Process (Dec-MDP) solved as a set of individual MDPs, where interactions between MDPs are considered in a distributed value function. Thus each robot computes locally a strategy that minimizes the interactions between the robots and maximizes the space coverage of the team. Our technique has been implemented and evaluated in real-world and simulated experiments.

This paper presents a distributed control architecture to perform part recognition and closed-loop control of a distributed manipulation device. This architecture is based on decentralized cells able to communicate with their four neighbors thanks to peer-to-peer links. Various original algorithms are proposed to reconstruct, recognize and convey the object levitating on a new contactless distributed manipulation device. Experimental results show that each algorithm does a good job for itself and that all the algorithms together succeed in sorting and conveying the objects to their final destination. In the future, this architecture may be used to control MEMS-arrayed manipulation surfaces in order to develop Smart Surfaces, for conveying, fine positioning and sorting of very small parts for micro-systems assembly lines.

Recently, a great deal of interest has been developed in learning in multi-agent systems to achieve decentralized control. Machine learning is a popular approach to find controllers that are tailored exactly to the system without any prior model. In this paper, we propose a semi-decentralized reinforcement learning control approach in order to position and convey an object on a contact-free MEMS-based distributed-manipulation system. The experimental results validate the semi-decentralized reinforcement learning method as a way to design control laws for such distributed systems.

Multi-agent systems (MAS) are a field of study of growing interest in a variety of domains such as robotics or distributed controls. The article focuses on decentralized reinforcement learning (RL) in cooperative MAS, where a team of independent learning robots (IL) try to coordinate their individual behavior to reach a coherent joint behavior. We assume that each robot has no information about its teammates' actions. To date, RL approaches for such ILs did not guarantee convergence to the optimal joint policy in scenarios where the coordination is difficult. We report an investigation of existing algorithms for the learning of coordination in cooperative MAS, and suggest a Q-learning extension for ILs, called hysteretic Q-learning. This algorithm does not require any additional communication between robots. Its advantages are showing off and compared to other methods on various applications: bi-matrix games, collaborative ball balancing task and pursuit domain.

Reinforcement learning (R-L) is an intuitive way of programming well-suited for use on autonomous robots because it does not need to specify how the task has to be achieved. However, RL remains difficult to implement in realistic domains because of its slowness in convergence. In this paper, we develop a theoretical study of the influence of some RL parameters over the learning speed. We also provide experimental justifications for choosing the reward function and initial Q-values in order to improve RL speed within the context of a goal-directed robot task