Recently, geospatial artificial intelligence (GeoAI) has risen as a set of essential technologies for urban mobility pattern mining and understanding. However, traditional deep learning models are constrained by their high data dependency and limited interpretability. This study introduces the Knowledge-Driven Semantic Tree (KD-ST) model, a novel GeoAI framework that integrates structured semantic descriptions with graph-based learning to enhance geospatial modeling on e-scooter ridership classification. By incorporating a street knowledge structure into the GeoAI model architecture, KD-ST bridges the gap between purely data-driven methods and knowledge-informed urban analytics, improving classification performance and model transparency. We conducted case studies in four major U.S. cities, including Austin, Phoenix, Denver, and Washington, D.C., to evaluate the proposed KD-ST model’s performance. The proposed model outperformed baseline models by 12.1% to 156.5% as for the F1 score. Moreover, to enhance transparency and reliability, key internal parameters were extracted to visualize and analyze the learned hierarchical knowledge structure. Results indicate that domain knowledge provides useful information for the design of deep learning models and improves model performance. Furthermore, the model achieved higher transferability among cities with more similar urban contexts, which provides valuable insights for e-scooter planners on model choice.

Tracking and manipulating irregularly-shaped, previously unseen objects in dynamic environments is important for robotic applications in manufacturing, assembly, and logistics. Recently introduced Gaussian Splats [1] efficiently model object geometry, but lack persistent state estimation for taskoriented manipulation. We present Persistent Object Gaussian Splat (POGS), a system that embeds semantics, self-supervised visual features, and object grouping features into a compact representation that can be continuously updated to estimate the pose of scanned objects. POGS updates object states without requiring expensive rescanning or prior CAD models of objects. After an initial multi-view scene capture and training phase, POGS uses a single stereo camera to integrate depth estimates along with self-supervised vision encoder features for object pose estimation. POGS supports grasping, reorientation, and natural language-driven manipulation by refining object pose estimates, facilitating sequential object reset operations with human-induced object perturbations and tool servoing, where robots recover tool pose despite tool perturbations of up to 30°. POGS achieves up to 12 consecutive successful object resets and recovers from 80% of in-grasp tool perturbations.

Building semantic 3D maps is valuable for searching for objects of interest in offices, warehouses, stores, and homes. We present a mapping system that incrementally builds a Language-Embedded Gaussian Splat (LEGS): a detailed 3D scene representation that encodes both appearance and semantics in a unified representation. LEGS is trained online as a robot traverses its environment to enable localization of open-vocabulary object queries. We evaluate LEGS on 4 room-scale scenes where we query for objects in the scene to assess how LEGS can capture semantic meaning. We compare LEGS to LERF [1] and find that while both systems have comparable object query success rates, LEGS trains over 3.5x faster than LERF. Results suggest that a multi-camera setup and incremental bundle adjustment can boost visual reconstruction quality in constrained robot trajectories, and suggest LEGS can localize open-vocabulary and long-tail object queries with up to 66% accuracy. See project website at: berkeleyautomation.github.io/LEGS

We propose a modular systems architecture, MANIP, that can facilitate the design and development of robot manipulation systems by systematically combining learned subpolicies with well-established procedural algorithmic primitives such as Inverse Kinematics, Kalman Filters, RANSAC outlier rejection, PID modules, etc. (aka "Good Old Fashioned Engineering (GOFE)"). The MANIP architecture grew from our lab’s experience developing robot systems for folding clothes, routing cables, and untangling knots. To address failure modes, MANIP can facilitate inclusion of "interactive perception" subpolicies that execute robot actions to modify system state to bring the system into alignment with the training distribution and / or to disambiguate system state when system state confidence is low. We demonstrate how MANIP can be applied with 3 case studies and then describe a detailed case study in cable tracing with experiments that suggest MANIP can improve performance by up to 88%. Code and details are available at: https://berkeleyautomation.github.io/MANIP/

The Internet of Things (IoT) offers transformative potential when combined with Machine Learning (ML), but labeling diverse IoT data remains challenging. To address this, we introduce SenseScape Testbed, an IoT experimentation platform for indoor environments with wireless sensor nodes, robots, and location-tracking nodes. This testbed enables IoT applications such as human activity recognition and indoor mobility tracking, supporting energy efficiency, occupant comfort, and context representation while providing a versatile environment for labeling and testing to advance ML algorithms tailored for IoT applications.

Effectively rearranging heterogeneous objects constitutes a high-utility skill that an intelligent robot should master. Whereas significant work has been devoted to the grasp synthesis of heterogeneous objects, little attention has been given to the planning for sequentially manipulating such objects. In this work, we examine the long-horizon sequential rearrangement of heterogeneous objects in a tabletop setting, addressing not just generating feasible plans but near-optimal ones. Toward that end, and building on previous methods, including combinatorial algorithms and Monte Carlo tree search-based solutions, we develop state-of-the-art solvers for optimizing two practical objective functions considering key object properties such as size and weight. Thorough simulation studies show that our methods provide significant advantages in handling challenging heterogeneous object rearrangement problems, especially in cluttered settings. Real robot experiments further demonstrate and confirm these advantages. Source code and evaluation data associated with this research will be available at https//github.com/arc-l/TRLB upon the publication of this manuscript

Exploration in reinforcement learning is, in general, a challenging problem. A common technique to make learning easier is providing demonstrations from a human supervisor, but such demonstrations can be expensive and time-consuming to acquire. In this work, we study a more tractable class of reinforcement learning problems defined simply by examples of successful outcome states, which can be much easier to provide while still making the exploration problem more tractable. In this problem setting, the reward function can be obtained automatically by training a classifier to categorize states as successful or not. However, as we will show, this requires the classifier to make uncertainty-aware predictions that are very difficult using standard techniques for training deep networks. To address this, we propose a novel mechanism for obtaining calibrated uncertainty based on an amortized technique for computing the normalized maximum likelihood (NML) distribution, leveraging tools from meta-learning to make this distribution tractable. We show that the resulting algorithm has a number of intriguing connections to both count-based exploration methods and prior algorithms for learning reward functions, while also providing more effective guidance towards the goal. We demonstrate that our algorithm solves a number of challenging navigation and robotic manipulation tasks which prove difficult or impossible for prior methods.

Reinforcement Learning (RL) algorithms can in principle acquire complex robotic skills by learning from large amounts of data in the real world, collected via trial and error. However, most RL algorithms use a carefully engineered setup in order to collect data, requiring human supervision and intervention to provide episodic resets. This is particularly evident in challenging robotics problems, such as dexterous manipulation. To make data collection scalable, such applications require reset-free algorithms that are able to learn autonomously, without explicit instrumentation or human intervention. Most prior work in this area handles single-task learning. However, we might also want robots that can perform large repertoires of skills. At first, this would appear to only make the problem harder. However, the key observation we make in this work is that an appropriately chosen multi-task RL setting actually alleviates the reset-free learning challenge, with minimal additional machinery required. In effect, solving a multi-task problem can directly solve the reset-free problem since different combinations of tasks can serve to perform resets for other tasks. By learning multiple tasks together and appropriately sequencing them, we can effectively learn all of the tasks together reset-free. This type of multi-task learning can effectively scale reset-free learning schemes to much more complex problems, as we demonstrate in our experiments. We propose a simple scheme for multi-task learning that tackles the reset-free learning problem, and show its effectiveness at learning to solve complex dexterous manipulation tasks in both hardware and simulation without any explicit resets. This work shows the ability to learn in-hand manipulation behaviors in the real world with RL without any human intervention.

The COVID-19 pandemic has impacted academic life in different ways. In the mobile and pervasive computing community, there was a struggle on data collection for the evaluation of human-sensing systems. An automated and contactless solution to collect data from users at home is one way that can help in the continuation of user-centric studies. In this poster, we present a portable system for remote, in-home data collection. The system is powered by a Raspberry Pi© and input peripherals (a camera, a microphone, and a wireless receiver). Our system uses a speech interface for text-to-speech and speech-to-text conversions. The system acts as a voice-based "smart agent" that guides the user during an experiment session. We aim to use our system to collect data from a set of smart pill bottles that we previously designed for medication adherence monitoring [1] and user identification [3].

The past 40 years have seen an explosion of Radio Frequency (RF) transmitters, which motivates understanding their impacts on the natural world. The European honeybee, Apis Mellifera, has been shown to sense the Earth's magnetic field. Human Radio Frequency (RF) transmitters alter this field. For example, recent work demonstrated that human-created RF interferes with the common robin's ability to orient themselves. This work proposes an experimental design to determine if honeybees can sense RF transmissions in frequencies from 1 MHz (AM radio) to 6 GHz (WiFi). We deployed a custom-designed RF bee feeder near bee hives to test honeybees' RF sensing ability.