Semantic maps are fundamental for robotics tasks such as navigation and manipulation. They also enable yield prediction and phenotyping in agricultural settings. In this paper, we introduce an efficient and scalable approach for active semantic mapping in horticultural environments, employing a mobile robot manipulator equipped with an RGB-D camera. Our method leverages probabilistic semantic maps to detect semantic targets, generate candidate viewpoints, and compute the corresponding information gain. We present an efficient ray-casting strategy and a novel information utility function that accounts for both semantics and occlusions. The proposed approach reduces total runtime by 8 % compared to previous baselines. Furthermore, our information metric surpasses other metrics in reducing multiclass entropy and improving surface coverage, particularly in the presence of segmentation noise. Real-world experiments validate our method's effectiveness but also reveal challenges such as depth sensor noise and varying environmental conditions, requiring further research. https://github.com/jrcuaranv/nbv_planning.

Due to labor shortages in specialty crop industries, a need for robotic automation to increase agricultural efficiency and productivity has arisen. Previous manipulation systems harvest well in uncluttered and structured environments. High tunnel environments are more compact and cluttered in nature, requiring a rethinking of the large form factor systems and grippers. We propose a novel co-designed framework incorporating a global detection camera and a local eye-in-hand camera that demonstrates precise localization of small fruits via closed-loop visual feedback and reliable error handling. Field experiments in high tunnels show that our system can reach 85.0% of cherry tomato fruit in 10.98s on average.

A neural network-based framework is developed and experimentally demonstrated for the problem of estimating the shape of a soft continuum arm (SCA) from noisy measurements of the pose at a finite number of locations along the length of the arm. The neural network takes as input these measurements and produces as output a finitedimensional approximation of the strain, which is further used to reconstruct the infinite-dimensional smooth posture. This problem is important for various soft robotic applications. It is challenging due to the flexible aspects that lead to the infinitedimensional reconstruction problem for the continuous posture and strains. Because of this, past solutions to this problem are computationally intensive. The proposed fast smooth reconstruction method is shown to be five orders of magnitude faster while having comparable accuracy. The framework is evaluated on two testbeds: a simulated octopus muscular arm and a physical BR2 pneumatic soft manipulator.

In visual Reinforcement Learning (RL), learning from pixel-based observations poses significant challenges on sample efficiency, primarily due to the complexity of extracting informative state representations from high-dimensional data. Previous methods such as contrastive-based approaches have made strides in improving sample efficiency but fall short in modeling the nuanced evolution of states. To address this, we introduce MOOSS, a novelframe-work that leverages a temporal contrastive objective with the help of graph-based spatial-temporal masking to explicitly model state evolution in visual RL. Specifically, we propose a self-supervised dual-component strategy that integrates (1) a graph construction of pixel-based observations for spatial-temporal masking, coupled with (2) a multilevel contrastive learning mechanism that enriches state representations by emphasizing temporal continuity and change of states. MOOSS advances the understanding of state dynamics by disrupting and learning from spatial-temporal correlations, which facilitates policy learning. Our comprehensive evaluation on multiple continuous and discrete control benchmarks shows that MOOSS outperforms previous state-of-the-art visual RL methods in terms of sample efficiency, demonstrating the effectiveness of our method.

Traffic forecasting, crucial for urban planning, requires accurate predictions of spatial-temporal traffic patterns across urban areas. Existing research mainly focuses on designing complex spatial-temporal models to capture these dependencies. However, this field faces challenges related to data scarcity and model stability, which results in limited performance improvement. To address these issues, we propose Spatial-Temporal Masked AutoEncoders (STMAE), a plug-and-play framework designed to enhance existing spatial-temporal models on traffic prediction. STMAE operates in two stages. In the pretraining stage, an encoder processes partially visible traffic data produced by a dual-masking strategy, including biased random walk-based spatial masking and patch-based temporal masking. Subsequently, two decoders aim to reconstruct the masked counterparts from both spatial and temporal perspectives. The fine-tuning stage retains the pretrained encoder and integrates it with decoders from existing backbones to improve traffic forecasting accuracy. Our results on traffic benchmarks show that STMAE can largely enhance the forecasting capabilities of various spatial-temporal models.

Accurate and robust navigation in unstructured environments requires fusing data from multiple sensors. Such fusion ensures that the robot is better aware of its surroundings, including areas of the environment that are not immediately visible but were visible at a different time. To solve this problem, we propose a method for traversability prediction in challenging outdoor environments using a sequence of RGB and depth images fused with pose estimations. Our method, termed WayFASTER (Waypoints-Free Autonomous System for Traversability with Enhanced Robustness), uses experience data recorded from a receding horizon estimator to train a self-supervised neural network for traversability prediction, eliminating the need for heuristics. Our experiments demonstrate that our method excels at avoiding obstacles, and correctly detects that traversable terrains, such as tall grass, can be navigable. By using a sequence of images, WayFASTER significantly enhances the robot’s awareness of its surroundings, enabling it to predict the traversability of terrains that are not immediately visible. This enhanced awareness contributes to better navigation performance in environments where such predictive capabilities are essential.

In the recent progress in embodied navigation and sim-to-robot transfer, modular policies have emerged as a de facto framework. However, there is more to compositionality beyond the decomposition of the learning load into modular components. In this work, we investigate a principled way to syntactically combine these components. Particularly, we propose Exploitation-Guided Exploration (XgX) where separate modules for exploration and exploitation come together in a novel and intuitive manner. We configure the exploitation module to take over in the deterministic final steps of navigation i.e. when the goal becomes visible. Crucially, an exploitation module teacher-forces the exploration module and continues driving an overridden policy optimization. XgX, with effective decomposition and novel guidance, improves the state-of-the-art performance on the challenging object navigation task from 70% to 73%. Along with better accuracy, through targeted analysis, we show that XgX is also more efficient at goal-conditioned exploration. Finally, we show sim-to-real transfer to robot hardware and XgX performs over two-fold better than the best baseline from simulation benchmarking. Project page: xgxvisnav.github.io

We propose the new task open-world video instance segmentation and captioning. It requires to detect, segment, track and describe with rich captions never before seen objects. This challenging task can be addressed by developing "abstractors" which connect a vision model and a language foundation model. Concretely, we connect a multi-scale visual feature extractor and a large language model (LLM) by developing an object abstractor and an object-to-text abstractor. The object abstractor, consisting of a prompt encoder and transformer blocks, introduces spatially-diverse open-world object queries to discover never before seen objects in videos. An inter-query contrastive loss further encourages the diversity of object queries. The object-to-text abstractor is augmented with masked cross-attention and acts as a bridge between the object queries and a frozen LLM to generate rich and descriptive object-centric captions for each detected object. Our generalized approach surpasses the baseline that jointly addresses the tasks of open-world video instance segmentation and dense video object captioning by 13% on never before seen objects, and by 10% on object-centric captions.

Episodic memory enables animals to map contexts and environmental features in space and time but is underused in artificial intelligence (AI). Here we show how simple associative learning rules can be expanded to basic episodic memory in AI. We augment an agent-based foraging simulation, ASIMOV, modeled on the simple neuronal circuitry of an invertebrate forager, by adding a novel computational module for simple episodic memory, the Feature Association Matrix (FAM). The FAM is a set of computationally light, graph learning algorithms which functionally resemble the auto- and hetero-associative circuits of the hippocampus for episodic memory. In simulations, FAM enables highly efficient foraging and navigation and shows how higher-order conditioning mechanisms give rise to spatial cognitive mapping by chaining pair-wise associations and encoding them with additional contexts. Thus, FAM demonstrates a biologically inspired, bottom-up enhancement of AI for higher-order cognition.

Object queries have emerged as a powerful abstraction to generically represent object proposals. However, their use for temporal tasks like video segmentation poses two questions: 1) How to process frames sequentially and propagate object queries seamlessly across frames. Using independent object queries per frame doesn't permit tracking, and requires post-processing. 2) How to produce temporally consistent, yet expressive object queries that model both appearance and position changes. Using the entire video at once doesn't capture position changes and doesn't scale to long videos. As one answer to both questions we propose 'context-aware relative object queries', which are continuously propagated frame-by-frame. They seamlessly track objects and deal with occlusion and re-appearance of objects, without post-processing. Further, we find context-aware relative object queries better capture position changes of objects in motion. We evaluate the proposed approach across three challenging tasks: video instance segmentation, multi-object tracking and segmentation, and video panoptic segmentation. Using the same approach and architecture, we match or surpass state-of-the art results on the diverse and challenging OVIS, Youtube-VIS, Cityscapes-VPS, MOTS 2020 and KITTI-MOTS data.

Small robots that can operate under the plant canopy can enable new possibilities in agriculture. However, unlike larger autonomous tractors, autonomous navigation for such under canopy robots remains an open challenge because Global Navigation Satellite System (GNSS) is unreliable under the plant canopy. We present a hybrid navigation system that autonomously switches between different sets of sensing modalities to enable full field navigation, both inside and outside of crop. By choosing the appropriate path reference source, the robot can accommodate for loss of GNSS signal quality and leverage row-crop structure to autonomously navigate. However, such switching can be tricky and difficult to execute over scale. Our system provides a solution by automatically switching between an exteroceptive sensing based system, such as Light Detection And Ranging (LiDAR) row-following navigation and waypoints path tracking. In addition, we show how our system can detect when the navigate fails and recover automatically extending the autonomous time and mitigating the necessity of human intervention. Our system shows an improvement of about 750 m per intervention over GNSS-based navigation and 500 m over row following navigation.

Node classification for graph-structured data aims to classify nodes whose labels are unknown. While studies on static graphs are prevalent, few studies have focused on dynamic graph node classification. Node classification on dynamic graphs is challenging for two reasons. First, the model needs to capture both structural and temporal information, particularly on dynamic graphs with a long history and require large receptive fields. Second, model scalability becomes a significant concern as the size of the dynamic graph increases. To address these problems, we propose the Time Augmented Dynamic Graph Neural Network (TADGNN) framework. TADGNN consists of two modules: 1) a time augmentation module that captures the temporal evolution of nodes across time structurally, creating a time-augmented spatio-temporal graph, and 2) an information propagation module that learns the dynamic representations for each node across time using the constructed time-augmented graph. We perform node classification experiments on four dynamic graph benchmarks. Experimental results demonstrate that TADGNN framework outperforms several static and dynamic state-of-the-art (SOTA) GNN models while demonstrating superior scalability. We also conduct theoretical and empirical analyses to validate the efficiency of the proposed method.

This work presents advances in predictive modeling of weed growth, as well as an improved planning index to be used in conjunction with these techniques, for the purpose of improving the performance of coordinated weeding algorithms being developed for industrial agriculture. We demonstrate that the evolving Gaussian process (E-GP) method applied to measurements from the agents can predict the evolution of the field within the realistic simulation environment, Weed World. This method also provides physical insight into the seed bank distribution of the field. In this work, we extend the E-GP model in two important ways. First, we have developed a model that has a bias term, and we show how it is connected to the seed bank distribution. Second, we show that one may decouple the component of the model representing weed growth from the component, which varies with the seed bank distribution, and adapt the latter online. We compare this predictive approach with one that relies on known properties of the weed growth model and show that the E-GP method can drive down the total weed biomass for fields with high seed bank densities using less agents, without assuming this model information. We use an improved planning index, the Whittle index, which allows a balanced tradeoff between exploiting a row or allowing it to accrue reward and conforms to what we show is the theoretical limit for the fewest number of agents, which can be used in this domain.

Multi-object tracking and segmentation (MOTS) is important for understanding dynamic scenes in video data. Existing methods perform well on multi-object detection and segmentation for independent video frames, but tracking of objects over time remains a challenge. MOTS methods formulate tracking locally, i.e., frame-by-frame, leading to sub-optimal results. Classical global methods on tracking operate directly on object detections, which leads to a combinatorial growth in the detection space. In contrast, we formulate a global method for MOTS over the space of assignments rather than detections: First, we find all top-k assignments of objects detected and segmented between any two consecutive frames and develop a structured prediction formulation to score assignment sequences across any number of consecutive frames. We use dynamic programming to find the global optimizer of this formulation in polynomial time. Second, we connect objects which reappear after having been out of view for some time. For this we formulate an assignment problem. On the challenging KITTI-MOTS and MOTSChallenge datasets, this achieves state-of-the-art results among methods which don’t use depth data.

We propose a novel scheme for surveillance of a dynamic ground convoy moving along a non-linear trajectory, by aerial agents that maintain a uniformly spaced formation on a time-varying elliptical orbit encompassing the convoy. Elliptical orbits are used as they are more economical than circular orbits for circumnavigating the group of targets in the moving convoy. The proposed scheme includes an algorithm for computing feasible elliptical orbits, a vector guidance law for agent motion along the desired orbit, and a cooperative strategy to control the speeds of the aerial agents in order to quickly achieve and maintain the desired formation. It achieves mission objectives while accounting for linear and angular speed constraints on the aerial agents. The scheme is validated through simulations and actual experiments with a convoy of ground robots and a team of quadrotors as the aerial agents, in a motion capture environment.

In this paper, we present a hybrid direct-indirect model reference adaptive controller (MRAC), to address a class of problems with matched and unmatched uncertainties. In the proposed architecture, the unmatched uncertainty is estimated online through a companion observer model. Upon convergence of the observer, the unmatched uncertainty estimate is remodeled into a state dependent linear form to augment the nominal system dynamics. Meanwhile, a direct adaptive controller designed for a switching system cancels the effect of matched uncertainty in the system and achieves reference model tracking. We demonstrate that the proposed hybrid controller can handle a broad class of nonlinear systems with both matched and unmatched uncertainties.

Soft robots undergo large nonlinear spatial deformations due to both inherent actuation and external loading. The physics underlying these deformations is complex, and often requires intricate analytical and numerical models. The complexity of these models may render traditional model-based control difficult and unsuitable. Model-free methods offer an alternative for analyzing the behavior of such complex systems without the need for elaborate modeling techniques. In this paper, we present a model-free approach for open loop position control of a soft spatial continuum arm, based on deep reinforcement learning. The continuum arm is pneumatically actuated and attains a spatial work-space by a combination of unidirectional bending and bidirectional torsional deformation. We use Deep-Q Learning with experience replay to train the system in simulation. The efficacy and robustness of the control policy obtained from the system is validated both in simulation and on the continuum arm prototype for varying external loading conditions.

In this paper, we present a new approach to transfer in Reinforcement Learning (RL) for cross-domain tasks. Unlike, available transfer approaches, where target task learning is accelerated through initialized learning from source, we propose to adapt and reuse the optimal source policy directly in the related domains. We show the optimal policy from a related source task can be near optimal in target domain provided an adaptive policy accounts for the model error between target and the projected source. A significant advantage of the proposed policy augmentation is in generalizing the policies across related domains without having to re-Iearn the new tasks. We demonstrate that, this architecture leads to better sample efficiency in the transfer, reducing sample complexity of target task learning to target apprentice learning.

We explore beyond existing work in learning from demonstration by asking the question: “Can robots learn to guide?”, that is, can a robot autonomously learn an instructional policy from expert demonstration and use it to instruct humans in executing complex task? As a solution, we propose learning of instructional policy (πI) that maps the state to an instruction for a human. To learn πI, we define action primitives that addresses the challenge of mapping continuous state action trajectories to human parse-able instructions. Action primitives are demonstrated to be very effective in automatic segmentation of demonstration trajectories into fewer repetitive and reusable segments, and a highly scalable approach in comparison to the existing state-of-the art. Finally, we construct a non-generic policy model as a generative model for instructional policies to generate instruction for an entire task. With few modifications, the proposed model is demonstrated to perform autonomous execution of complex truck loading task on hydraulic actuated scaled excavator robot. Guidance approach is tested based on a controlled group study involving 75 participants, who learn to perform the same task.

This work presents a strategy for coordinated multi-agent weeding under conditions of partial environmental information. The goal of this work is to demonstrate the feasibility of coordination strategies for improving the weeding performance of autonomous agricultural robots. We show that, given a sufficient number of agents, the algorithm can successfully weed fields with various initial seed bank densities, even when multiple days are allowed to elapse before weeding commences. Furthermore, the use of coordination between agents is demonstrated to strongly improve system performance as the number of agents increases, enabling the system to eliminate all the weeds in the field, as in the case of full environmental information, when the planner without coordination failed to do so. As a domain to test our algorithms, we have developed an open source simulation environment, Weed World, which allows real-time visualization of coordinated weeding policies, and includes realistic weed generation. In this work, experiments are conducted to determine the required number of agents and their required transit speed, for given initial seed bank densities and varying allowed days before the start of the weeding process.

Combining the benefits of robust situational awareness of human operators with the efficiency and precision of automatic control has been an important topic of human-machine shared control. The emphasis is on keeping human operators in the loop while automatic control providing assistance to improve task performance. Given a task with specific subgoals, execution of a task using blended shared control involves predicting the operator's intent of subgoal transitions and deciding the blending weights for inputs from the human operator and automatic control. In this paper we address the problem of subgoal adjustment in blended shared control which is typically initiated by the operator's intent and necessary to sustain the shared control performance for changing subgoal conditions. First, we provide a method to predict operator's intent of visiting a subgoal. Based on intent prediction, we propose a method for subgoal adjustment where the adjustment is encoded by a hyperrectangle. The volume of the hyper-rectangle is obtained by using a hyperbolic slope transition function which is based on the distance between subgoals. The adjustment actions within the hyper-rectangle are facilitated by a skill-weighted action integral that takes into consideration the skill level of the operator. The approach is tested on a scaled hydraulic excavator platform with multiple novice operators and a skilled operator. Experimental results are presented and discussed.

Many prediction, decision-making, and control architectures rely on online learned Gaussian process (GP) models. However, most existing GP regression algorithms assume a single generative model, leading to poor predictive performance when the data are nonstationary, i.e., generated from multiple switching processes. Furthermore, existing methods for GP regression over nonstationary data require significant computation, do not come with provable guarantees on correctness and speed, and many only work in batch settings, making them ill-suited for real-time prediction. We present an efficient online GP framework, GP-non-Bayesian clustering (GP-NBC), which addresses these computational and theoretical issues, allowing for real-time changepoint detection and regression using GPs. Our empirical results on two real-world data sets and two synthetic data set show that GP-NBC outperforms state-of-the-art methods for nonstationary regression in terms of both regression error and computation. For example, it outperforms Dirichlet process GP clustering with Gibbs sampling by 98% in computation time reduction while the mean absolute error is comparable.

We consider the problem of estimating the latent state of a spatiotemporally evolving continuous function using very few sensor measurements. We show that layering a dynamical systems prior over temporal evolution of weights of a kernel model is a valid approach to spatiotemporal modeling that does not necessarily require the design of complex nonstationary kernels. Furthermore, we show that such a predictive model can be utilized to determine sensing locations that guarantee that the hidden state of the phenomena can be recovered with very few measurements. We provide sufficient conditions on the number and spatial location of samples required to guarantee state recovery, and provide a lower bound on the minimum number of samples required to robustly infer the hidden states. Our approach outperforms existing methods in numerical experiments.

Most current model reference adaptive control (MRAC) methods rely on parametric adaptive elements, in which the number of parameters of the adaptive element are fixed a priori, often through expert judgment. An example of such an adaptive element is radial basis function networks (RBFNs), with RBF centers preallocated based on the expected operating domain. If the system operates outside of the expected operating domain, this adaptive element can become noneffective in capturing and canceling the uncertainty, thus rendering the adaptive controller only semiglobal in nature. This paper investigates a Gaussian process-based Bayesian MRAC architecture (GP-MRAC), which leverages the power and flexibility of GP Bayesian nonparametric models of uncertainty. The GP-MRAC does not require the centers to be preallocated, can inherently handle measurement noise, and enables MRAC to handle a broader set of uncertainties, including those that are defined as distributions over functions. We use stochastic stability arguments to show that GP-MRAC guarantees good closed-loop performance with no prior domain knowledge of the uncertainty. Online implementable GP inference methods are compared in numerical simulations against RBFN-MRAC with preallocated centers and are shown to provide better tracking and improved long-term learning.

A growing concern with deploying Unmanned Aerial Vehicles (UAVs) in urban environments is the potential violation of human privacy, and the backlash this could entail. Therefore, there is a need for UAV path planning algorithms that minimize the likelihood of invading human privacy. We formulate the problem of human-aware path planning as a nonstationary Markov Decision Process, and provide a novel model-based reinforcement learning solution that leverages Gaussian process clustering. Our algorithm is flexible enough to accommodate changes in human population densities by employing Bayesian nonparametrics, and is real-time computable. The approach is validated experimentally on a large-scale long duration experiment with both simulated and real UAVs.

Commonly used Proportional-Integral-Derivative based UAV flight controllers are often seen to provide adequate trajectory-tracking performance, but only after extensive tuning. The gains of these controllers are tuned to particular platforms, which makes transferring controllers from one UAV to other time-intensive. This paper formulates the problem of control-transfer from a source system to a transfer system and proposes a solution that leverages well-studied techniques in adaptive control. It is shown that concurrent learning adaptive controllers improve the trajectory tracking performance of a quadrotor with the baseline linear controller directly imported from another quadrotor whose inertial characteristics and throttle mapping are very different. Extensive flight-testing, using indoor quadrotor platforms operated in MIT's RAVEN environment, is used to validate the method.

Classical work in model reference adaptive control for uncertain nonlinear dynamical systems with a radial basis function (RBF) neural network adaptive element does not guarantee that the network weights stay bounded in a compact neighborhood of the ideal weights when the system signals are not persistently exciting (PE). Recent work has shown, however, that an adaptive controller using specifically recorded data concurrently with instantaneous data guarantees boundedness without PE signals. However, the work assumes fixed RBF network centers, which requires domain knowledge of the uncertainty. Motivated by reproducing kernel Hilbert space theory, we propose an online algorithm for updating the RBF centers to remove the assumption. In addition to proving boundedness of the resulting neuro-adaptive controller, a connection is made between PE signals and kernel methods. Simulation results show improved performance.