The study of non-line-of-sight (NLOS) imaging is growing due to its many potential applications, including rescue operations and pedestrian detection by self-driving cars. However, implementing NLOS imaging on a moving camera remains an open area of research. Existing NLOS imaging methods rely on time-resolved detectors and laser configurations that require precise optical alignment, making it difficult to deploy them in dynamic environments. This work proposes a data-driven approach to NLOS imaging, PathFinder, that can be used with a standard RGB camera mounted on a small, power-constrained mobile robot, such as an aerial drone. Our experimental pipeline is designed to accurately estimate the 2D trajectory of a person who moves in a Manhattan-world environment while remaining hidden from the camera’s field-of-view. We introduce a novel approach to process a sequence of dynamic successive frames in a line-of-sight (LOS) video using an attention-based neural network that performs inference in real-time. The method also includes a preprocessing selection metric that analyzes images from a moving camera which contain multiple vertical planar surfaces, such as walls and building facades, and extracts planes that return maximum NLOS information. We validate the approach on in-the-wild scenes using a drone for video capture, thus demonstrating low-cost NLOS imaging in dynamic capture environments. The real-world dataset that we collected and used to train the network can be found at https://srchandr.github.io/DynamicNLOS/.

In recent years, small-scale driving testbeds have been developed as controlled physical environments for the evaluation of autonomous vehicle controllers. Such controllers are heavily dependent on computer vision algorithms that enable the vehicle to perceive its surroundings. To bridge the Sim2Real content and appearance gap between simulated and real-world image data for training these algorithms, we propose a novel transfer learning approach that performs domain adaptation using StyleGAN to generate style-mixed images that closely resemble real-world images. We explain our approach within the context of our small-scale driving testbed, CHARTOPOLIS, and demonstrate it on synthetic image data of two object classes, vehicles and buildings, from the driving simulator CARLA. Our results show that this approach works on the vehicle object class while failing on the building object class. This paper thus provides a plausible approach to bridging the Sim2Real gap through the use of custom pipelines that augment image datasets using a mix of techniques for domain adaptation and domain randomization.

The problem of adapting models from a source domain using data from any target domain of interest has gained prominence, thanks to the brittle generalization in deep neural networks. While several test-time adaptation techniques have emerged, they typically rely on synthetic data augmentations in cases of limited target data availability. In this paper, we consider the challenging setting of single-shot adaptation and explore the design of augmentation strategies. We argue that augmentations utilized by existing methods are insufficient to handle large distribution shifts, and hence propose a new approach SiSTA (Single-Shot Target Augmentations), which first fine-tunes a generative model from the source domain using a single-shot target, and then employs novel sampling strategies for curating synthetic target data. Using experiments with a state-of-the-art domain adaptation method, we find that SiSTA produces improvements as high as 20% over existing baselines under challenging shifts in face attribute detection, and that it performs competitively to oracle models obtained by training on a larger target dataset. Our codes can be accessed at github.com/kowshikthopalli/SISTA.

Controlling soft continuum robotic arms is challenging due to their hyper-redundancy and dexterity. In this paper we experimentally demonstrate, for the first time, closed-loop control of the configuration space variables of a soft robotic arm, composed of independently controllable segments, using a Cosserat rod model of the robot and the distributed sensing and actuation capabilities of the segments. Our controller solves the inverse dynamic problem by simulating the Cosserat rod model in MATLAB using a computationally efficient numerical solution scheme, and it applies the computed control output to the actual robot in real time. The position and orientation of the tip of each segment are measured in real time, while the remaining unknown variables that are needed to solve the inverse dynamics are estimated simultaneously in the simulation. We implement the controller on a multi-segment silicone robotic arm with pneumatic actuation, using a motion capture system to measure the segments' positions and orientations. The controller is used to reshape the arm into configurations that are achieved through combinations of bending and extension deformations in 3D space. Although the possible deformations are limited for this robot platform, our study demonstrates the potential for implementing the control approach on a wide range of continuum robots in practice. The resulting tracking performance indicates the effectiveness of the controller and the accuracy of the simulated Cosserat rod model.

Robust feature matching forms the backbone for most Visual Simultaneous Localization and Mapping (vSLAM), visual odometry, 3D reconstruction, and Structure from Motion (SfM) algorithms. However, recovering feature matches from texture-poor scenes is a major challenge and still remains an open area of research. In this paper, we present a Stereo Visual Odometry (StereoVO) technique based on point and line features which uses a novel feature-matching mechanism based on an Attention Graph Neural Network that is designed to perform well even under adverse weather conditions such as fog, haze, rain, and snow, and dynamic lighting conditions such as nighttime illumination and glare scenarios. We perform experiments on multiple real and synthetic datasets to validate our method's ability to perform StereoVO under low-visibility weather and lighting conditions through robust point and line matches. The results demonstrate that our method achieves more line feature matches than state-of-the-art line-matching algorithms, which when complemented with point feature matches perform consistently well in adverse weather and dynamic lighting conditions.

In this paper, we present a decentralized control approach based on a Nonlinear Model Predictive Control (NMPC) method that employs barrier certificates for safe navigation of multiple nonholonomic wheeled mobile robots in unknown environments with static and/or dynamic obstacles. This method incorporates a Learned Barrier Function (LBF) into the NMPC design in order to guarantee safe robot navigation, i.e., prevent robot collisions with other robots and the obstacles. We refer to our proposed control approach as NMPC-LBF. Since each robot does not have a priori knowledge about the obstacles and other robots, we use a Deep Neural Network (DeepNN) running in real-time on each robot to learn the Barrier Function (BF) only from the robot's LiDAR and odometry measurements. The DeepNN is trained to learn the BF that separates safe and unsafe regions. We implemented our proposed method on simulated and actual Turtlebot3 Burger robot(s) in different scenarios. The implementation results show the effectiveness of the NMPC-LBF method at ensuring safe navigation of the robots.

Recovering rigid registration between successive camera poses lies at the heart of 3D reconstruction, SLAM and visual odometry. Registration relies on the ability to compute discriminative 2D features in successive camera images for determining feature correspondences, which is very challenging in feature-poor environments, i.e. low-texture and/or low-light environments. In this paper, we aim to address the challenge of recovering rigid registration between successive camera poses in feature-poor environments in a Visual Inertial Odometry (VIO) setting. In addition to inertial sensing, we instrument a small aerial robot with an RGBD camera and propose a framework that unifies the incorporation of 3D geometric entities: points, lines, and planes. The tracked 3D geometric entities provide constraints in an Extended Kalman Filtering framework. We show that by directly exploiting 3D geometric entities, we can achieve improved registration. We demonstrate our approach on different texture-poor environments, with some containing only flat texture-less surfaces providing essentially no 2D features for tracking. In addition, we evaluate how the addition of different 3D geometric entities contributes to improved pose estimation by comparing an estimated pose trajectory to a ground truth pose trajectory obtained from a motion capture system. We consider computationally efficient methods for detecting 3D points, lines and planes, since our goal is to implement our approach on small mobile robots, such as drones.

In this article, we introduce a model and a control approach for herding a swarm of “follower” agents to a target distribution among a set of states using a single “leader” agent. The follower agents evolve on a finite state space that is represented by a graph and transition between states according to a continuous-time Markov chain (CTMC), whose transition rates are determined by the location of the leader agent. The control problem is to define a sequence of states for the leader agent that steers the probability density of the forward equation of the Markov chain. For the case, when the followers are possibly interacting, we prove local approximate controllability of the system about equilibrium probability distributions. For the case, when the followers are noninteracting, we design two switching control laws for the leader that drive the swarm of follower agents asymptotically to a target probability distribution that is positive for all states. The first strategy is open-loop in nature, and the switching times of the leader are independent of the follower distribution. The second strategy is of feedback type, and the switching times of the leader are functions of the follower density in the leader's current state. We validate our control approach through numerical simulations with varied numbers of follower agents that evolve on graphs of different sizes, through a 3-D multirobot simulation in which a quadrotor is used to control the spatial distribution of eight ground robots over four regions, and through a physical experiment in which a swarm of ten robots is herded by a virtual leader over four regions.

The Go-CHART is a four-wheel, skid-steer robot that resembles a 1:28 scale standard commercial sedan. It is equipped with an onboard sensor suite and both onboard and external computers that replicate many of the sensing and computation capabilities of a full-size autonomous vehicle. The Go-CHART can autonomously navigate a small-scale traffic testbed, responding to its sensor input wiwithth programmed controllers. Alternatively, it can be remotely driven by a user who views the testbed through the robot's four camera feeds, which facilitates safe, controlled experiments on driver interactions with driverless vehicles. We demonstrate the Go-CHART's ability to perform lane tracking and detection of traffic signs, traffic signals, and other Go-CHARTs in real-time, utilizing an external GPU that runs computationally intensive computer vision and deep learning algorithms.

In this paper, we present an optimal control approach using Linear Matrix Inequalities (LMIs) for trajectory tracking control of a three-wheeled omnidirectional mobile robot in the presence of external disturbances on the robot's actuators and noise in the robot's sensor measurements. First, a state-space representation of the omnidirectional robot dynamics is derived using a point-mass dynamic model. Then, we propose an LMI-based full-state feedback H∞-optimal controller for the tracking problem. The robot's tracking performance with the H∞-optimal controller is compared to its performance with a classical full-state feedback tracking controller in simulations with circular and bowtie-shaped reference trajectories. In order to evaluate our proposed controller in practice, we also implement the H∞-optimal and classical controllers for these reference trajectories on a three-wheeled omnidirectional robot. The H∞-optimal controller guarantees stabilization of the robot motion and attenuates the effects of frictional disturbances and measurement noise on the robot's tracking performance. Using the H∞-optimal controller, the robot is able to track the reference trajectories with up to a 47.8% and 45.8% decrease in the maximum pose and twist errors, respectively, over a full cycle of the trajectory compared to the classical controller. The simulation and experimental results show that our LMI-based H∞-optimal controller is robust to undesired effects of disturbances and noise on the dynamic behavior of the robot during trajectory tracking and can outperform the classical controller in attenuating their effects.

In this article, we present a novel distributed method for constructing an occupancy grid map of an unknown environment using a swarm of robots with global localization capabilities and limited interrobot communication. The robots explore the domain by performing Lévy walks in which their headings are defined by maximizing the mutual information between the robot's estimate of its environment in the form of an occupancy grid map and the distance measurements that it is likely to obtain when it moves in that direction. Each robot is equipped with laser range sensors, and it builds its occupancy grid map by repeatedly combining its own distance measurements with map information that is broadcast by neighboring robots. Using results on average consensus over time-varying graph topologies, we prove that all robots' maps will eventually converge to the actual map of the environment. In addition, we demonstrate that a technique based on topological data analysis, developed in our previous work for generating topological maps, can be readily extended for adaptive thresholding of occupancy grid maps. We validate the effectiveness of our distributed exploration and mapping strategy through a series of two-dimensional simulations and multirobot experiments.

This paper explores a stochastic approach for controlling swarms of independent robots toward a target distribution in a bounded domain. The robot swarm has no central controller, and individual robots lack both communication and localization capabilities. Robots can only measure a scalar field (e.g. concentration of a chemical) from the environment and from this deduce the desired local swarm density. Based on this value, each robot follows a simple control law that causes the swarm as a whole to diffuse toward the target distribution. Using a new holonomic drive robot, we present the first confirmation of this control law with physical experiment. Despite deviations from assumptions underpinning the theory, the swarm achieves the theorized convergence to the target distribution in both simulation and experiment. In fact, simulated and experimental performance agree with one another and with our hypothesis that the error from the target distribution is inversely proportional to the square root of the number of robots. This is evidence that the algorithm is both practical and easily scalable to large swarms.

This paper addresses a trajectory planning and task allocation problem for a swarm of resource-constrained robots that cannot localize or communicate with each other and that exhibit stochasticity in their motion and task-switching policies. We model the population dynamics of the robotic swarm as a set of advection-diffusion-reaction partial differential equations (PDEs), a linear parabolic PDE model that is bilinear in the robots' velocity and task-switching rates. These parameters constitute a set of time-dependent control variables that can be optimized and broadcast to the robots prior to their deployment. The planning and allocation problem can then be formulated as a PDE-constrained optimization problem, which we solve using techniques from optimal control. Simulations of a commercial pollination scenario validate the ability of our control approach to drive a robotic swarm to achieve predefined spatial distributions of activity over a closed domain, which may contain obstacles.

We present a novel analytical approach to computing the population and geometric parameters of a multi-robot system that will provably produce specified boundary coverage statistics. We consider scenarios in which robots with no global position information, communication, or prior environmental data have arrived at uniformly random locations along a simple closed or open boundary. This type of scenario can arise in a variety of multi-robot tasks, including surveillance, collective transport, disaster response, and therapeutic and imaging applications in nanomedicine. We derive the probability that a given point robot configuration is saturated, meaning that all pairs of adjacent robots are no farther apart than a specified distance. This derivation relies on a geometric interpretation of the saturation probability and an application of the Inclusion-Exclusion Principle, and it is easily extended to finite-sized robots. In the process, we obtain formulas for (a) an integral that is in general computationally expensive to compute directly, and (b) the volume of the intersection of a regular simplex with a hypercube. In addition, we use results from order statistics to compute the probability distributions of the robot positions along the boundary and the distances between adjacent robots. We validate our derivations of these probability distributions and the saturation probability using Monte Carlo simulations of scenarios with both point robots and finite-sized robots.

Collective food transport in ant colonies is a striking, albeit poorly understood, example of coordinated group behavior in nature that can serve as a template for robust, decentralized multi-robot cooperative manipulation strategies. We investigate this behavior in Aphaenogaster cockerelli ants in order to derive a model of the ants' roles and behavioral transitions and the resulting dynamics of a transported load. In experimental trials, A. cockerelli are induced to transport a rigid artificial load to their nest. From video recordings of the trials, we obtain time series data on the load position and the population counts of ants in three roles. From our observations, we develop a stochastic hybrid system model that describes the time evolution of these variables and that can be used to derive the dynamics of their statistical moments. In our model, ants switch stochastically between roles at constant, unknown probability rates, and ants in one role pull on the load with a force that acts as a proportional controller on the load velocity with unknown gain and set point. We compute these unknown parameters by using standard numerical optimization techniques to fit the time evolution of the means of the load position and population counts to the averaged experimental time series. The close fit of our model to the averaged data and to data for individual trials demonstrates the accuracy of our proposed model in predicting the ant behavior.

We compare our previously developed deterministic [7] and stochastic [3], [4] strategies for allocating tasks in robotic swarms1 consisting of very large populations of highly resource-constrained robots. We study our two task allocation approaches in a simulated scenario in which a collective of insect-inspired micro-aerial vehicles (MAVs) must produce a specified spatial distribution of pollination activity over a crop field. We investigate the approaches' requirements, advantages, and disadvantages under realistic conditions of error in robot localization, navigation, and sensing in simulation. Our results show that the deterministic approach, which requires region-based robot navigation, yields higher task progress in all cases. For robots without such navigation capabilities, the stochastic approach is a feasible alternative, and its resulting task progress is less sensitive to error in localization, error in navigation, and a combination of high error in localization, navigation, and sensing.

We present an approach to designing scalable, decentralized control policies that produce a desired collective behavior in a spatially inhomogeneous robotic swarm that emulates a system of chemically reacting molecules. Our approach is based on abstracting the swarm to an advection-diffusion-reaction partial differential equation model, which we solve numerically using smoothed particle hydrodynamics (SPH), a meshfree technique that is suitable for advection-dominated systems. The parameters of the macroscopic model are mapped onto the deterministic and random components of individual robot motion and the probabilities that determine stochastic robot task transitions. For very large swarms that are prohibitively expensive to simulate, the macroscopic model, which is independent of the population size, is a useful tool for synthesizing robot control policies with guarantees on performance in a top-down fashion. We illustrate our methodology by formulating a model of rabbiteye blueberry pollination by a swarm of robotic bees and using the macroscopic model to select control policies for efficient pollination.

We present a scalable approach to optimizing robot control policies for a target collective behavior in a spatially inhomogeneous robotic swarm. The approach can incorporate robot feedback to maintain system performance in an unknown environmental flow field. We consider systems in which the robots follow both deterministic and random motion and transition stochastically between tasks. Our methodology is based on an abstraction of the swarm to a macroscopic continuous model, whose dimensionality is independent of the population size, that describes the expected time evolution of swarm subpopulations over a discretization of the environment. We incorporate this model into a stochastic optimization method and map the optimized model parameters onto the robot motion and task transition control policies to achieve a desired global objective. We illustrate our methodology with a scenario in which the behaviors of a swarm of robotic bees are optimized for both uniform and nonuniform pollination of a blueberry field, including in the presence of an unknown wind.

Group food retrieval in some ant species serves as a useful paradigm for multirobot collective transport strategies that are decentralized, scalable, and do not require a priori information about the payload. We present a comprehensive overview of group retrieval in ants and investigate this phenomenon in Aphaenogaster cockerelli in order to extract the ants' roles during transport, the rules that govern their actions, and the individual forces that they apply to guide a food item to their nest. To measure these forces, we fabricated elastic structures with calibrated stiffness properties, induced ants to retrieve the structures, and tracked the resulting deformations with a camera. We then developed a hybrid system model of the ant behaviors that were observed in the experiments. We conducted simulations of the behavioral model that incorporate a quasi-static model of planar manipulation with compliant attachment points. Our simulations qualitatively replicate individual ant activity as well as certain macroscopic features of the transport.

We present a decentralized, scalable approach to assembling a group of heterogeneous parts into different products using a swarm of robots. While the assembly plans are predetermined, the exact sequence of assembly of parts and the allocation of subassembly tasks to robots are determined by the interactions between robots in a decentralized fashion in real time. Our approach is based on developing a continuous abstraction of the system derived from models of chemical reactions and formulating the strategy as a problem of selecting rates of assembly and disassembly. These rates are mapped onto probabilities that determine stochastic control policies for individual robots, which then produce the desired aggregate behavior. This top-down approach to determining robot controllers also allows us to optimize the rates at the abstract level to achieve fast convergence to the specified target numbers of products. Because the method incorporates programs for assembly and disassembly, changes in demand can lead to reconfiguration in a seamless fashion. We illustrate the methodology using a physics-based simulator with examples involving 15 robots and two types of final products.

We present a scalable approach to dynamically allocating a swarm of homogeneous robots to multiple tasks, which are to be performed in parallel, following a desired distribution. We employ a decentralized strategy that requires no communication among robots. It is based on the development of a continuous abstraction of the swarm obtained by modeling population fractions and defining the task allocation problem as the selection of rates of robot ingress and egress to and from each task. These rates are used to determine probabilities that define stochastic control policies for individual robots, which, in turn, produce the desired collective behavior. We address the problem of computing rates to achieve fast redistribution of the swarm subject to constraint(s) on switching between tasks at equilibrium. We present several formulations of this optimization problem that vary in the precedence constraints between tasks and in their dependence on the initial robot distribution. We use each formulation to optimize the rates for a scenario with four tasks and compare the resulting control policies using a simulation in which 250 robots redistribute themselves among four buildings to survey the perimeters.

We present a methodology for characterizing and synthesizing swarm behaviors using both a macroscopic model that represents a swarm as a continuum and a microscopic model that represents individual robots. We develop a systematic approach for synthesizing behaviors at the macroscopic level that can be realized on individual robots at the microscopic level. Our methodology is inspired by a dynamical model of ant house hunting [1], a decentralized process in which a colony attempts to emigrate to the best site among several alternatives. The model is hybrid because the colony switches between different sets of behaviors, or modes, during this process. At the macroscopic level, we are able to synthesize controllers that result in the deployment of a robotic swarm in a predefined ratio between distinct sites. We then derive hybrid controllers for individual robots using only local interactions and no communication that respect the specifications of the global continuous behavior. Our simulations demonstrate that our synthesis procedure yields a correct microscopic model from the macroscopic description with guarantees on performance at both levels

We present an approach for the dynamic assignment and reassignment of a large team of homogeneous robotic agents to multiple locations with applications to search and rescue, reconnaissance and exploration missions. Our work is inspired by experimental studies of ant house hunting and empirical models that predict the behavior of the colony that is faced with a choice between multiple candidate nests. We design stochastic control policies that enable the team of agents to distribute themselves between multiple candidate sites in a specified ratio. Additionally, we present an extension to our model to enable fast convergence via switching behaviors based on quorum sensing. The stability and convergence properties of these control policies are analyzed and simulation results are presented.