Robotic grasping is a cornerstone capability of embodied systems. Many methods directly output grasps from partial information without modeling the geometry of the scene, leading to suboptimal motion and even collisions. To address these issues, we introduce ZeroGrasp, a novel framework that simultaneously performs 3D reconstruction and grasp pose prediction in near real-time. A key insight of our method is that occlusion reasoning and modeling the spatial relationships between objects is beneficial for both accurate reconstruction and grasping. We couple our method with a novel large-scale synthetic dataset, which comprises 1M photo-realistic images, high-resolution 3D reconstructions and 11.3B physically-valid grasp pose annotations for 12K objects from the Objaverse-LVIS dataset. We evaluate Zero-Grasp on the GraspNet-1B benchmark as well as through real-world robot experiments. ZeroGrasp achieves state-of-the-art performance and generalizes to novel real-world objects by leveraging synthetic data. https://sh8.io/#/zerograsp

Fabric manipulation is a long-standing challenge in robotics due to the enormous state space and complex dynamics. Learning approaches stand out as promising for this domain as they allow us to learn behaviours directly from data. Most prior methods however rely heavily on simulation, which is still limited by the large sim-to-real gap of deformable objects or rely on large datasets. A promising alternative is to learn fabric manipulation directly from watching humans perform the task. In this work, we explore how demonstrations for fabric manipulation tasks can be collected directly by humans, providing an extremely natural and fast data collection pipeline. Then, using only a handful of such demonstrations, we show how a pick-and-place policy can be learned and deployed on a real robot, without any robot data collection at all. We demonstrate our approach on a fabric smoothing and folding task, showing that our policy can reliably reach folded states from crumpled initial configurations. Code, video and data are available on the project website: https://sites.google.com/view/foldingbyhand

This paper addresses the challenging problem of category-level pose estimation. Current state-of-the-art methods for this task face challenges when dealing with symmetric objects and when attempting to generalize to new environments solely through synthetic data training. In this work, we address these challenges by proposing a probabilistic model that relies on diffusion to estimate dense canonical maps crucial for recovering partial object shapes as well as establishing correspondences essential for pose estimation. Furthermore, we introduce critical components to enhance performance by leveraging the strength of the diffusion models with multi-modal input representations. We demonstrate the effectiveness of our method by testing it on a range of real datasets. Despite being trained solely on our generated synthetic data, our approach achieves state-of-the-art performance and unprecedented generalization qualities, outperforming baselines, even those specifically trained on the target domain. Our code and data for the generalization benchmark can be found at https://woven-planet.github.io/DiffusionNOCS/.

Learning-based grasp detectors typically assume a precision grasp, where each finger only has one contact point, and estimate the grasp probability. In this work, we propose a data generation and learning pipeline that can leverage power grasping, which has more contact points with an enveloping configuration and is robust against both positioning error and force disturbance. To train a grasp detector to prioritize power grasping while still keeping precision grasping as the secondary choice, we propose to train the network against the magnitude of disturbance in the gravity direction a grasp can resist (gravity-rejection score) rather than the binary classification of success. We also provide an efficient data generation pipeline for a dataset with gravity-rejection score annotation. Evaluation in both simulation and real-robot clarifies the significant improvement in our approach, especially when the objects are heavy.

The aim of this paper is to examine denoising autoencoders (DAEs) for improving the detection of right whales recorded in harsh marine environments. Passive acoustic recordings are taken from autonomous surface vehicles (ASVs) and are subject to noise from sources such as shipping and offshore construction. To mitigate the noise we apply DAEs and consider how best to train the classifier by augmenting clean training data with examples contaminated by noise. Evaluations find that the DAE improves detection accuracy and is particularly effective when the classifier is trained on data that has itself been denoised rather than using a clean model. Further, testing on unseen noises is also effective particularly for noises that exhibit similar character to noises seen in training.

Practical industrial assembly scenarios often require robotic agents to adapt their skills to unseen tasks quickly. While transfer reinforcement learning (RL) could enable such quick adaptation, much prior work has to collect many samples from source environments to learn target tasks in a model-free fashion, which still lacks sample efficiency on a practical level. In this work, we develop a novel transfer RL method named TRANSfer learning by Aggregating dynamics Models (TRANS-AM). TRANS-AM is based on model-based RL (MBRL) for its high-level sample efficiency, and only requires dynamics models to be collected from source environments. Specifically, it learns to aggregate source dynamics models adaptively in an MBRL loop to better fit the state-transition dynamics of target environments and execute optimal actions there. As a case study to show the effectiveness of this proposed approach, we address a challenging contact-rich peg-in-hole task with variable hole orientations using a soft robot. Our evaluations with both simulation and real-robot experiments demonstrate that TRANS-AM enables the soft robot to accomplish target tasks with fewer episodes compared when learning the tasks from scratch.

In industrial assembly tasks, the position of an object grasped by the robot has to be known with high precision in order to insert or place it. In real applications, this problem is commonly solved by jigs that are specially produced for each part. However, they significantly limit flexibility and are prohibitive when the target parts change often, so a flexible method to localize parts with high accuracy after grasping is desired. To solve this problem, we propose a method that can estimate the position of an object in the robot's hand to sub-millimeter precision, and can improve its estimate incrementally, using only minimal calibration and a force sensor. Our method is applicable to any robotic gripper and any rigid object that the gripper can hold, and requires only a force sensor. We demonstrate that the method can determine the position of an object to a precision of under 1 mm without using any part-specific jigs or equipment.

In this study, we present a novel control framework for assembly tasks with a soft robot. Typically, existing hard robots require high frequency controllers and precise force/torque sensors for assembly tasks. The resulting robot system is complex, entailing large amounts of engineering and maintenance. Physical softness allows the robot to interact with the environment easily. We expect soft robots to perform assembly tasks without the need for high frequency force/torque controllers and sensors. However, specific data-driven approaches are needed to deal with complex models involving nonlinearity and hysteresis. If we were to apply these approaches directly, we would be required to collect very large amounts of training data. To solve this problem, we argue that by leveraging softness and environmental constraints, a robot can complete tasks in lower dimensional state and action spaces, which could greatly facilitate the exploration of appropriate assembly skills. Then, we apply a highly efficient model-based reinforcement learning method to lower dimensional systems. To verify our method, we perform a simulation for peg-in-hole tasks. The results show that our method learns the appropriate skills faster than an approach that does not consider lower dimensional systems. Moreover, we demonstrate that our method works on a real robot equipped with a compliant module on the wrist.

Physical softness has been proposed to absorb impacts when establishing contact with a robot or its workpiece, to relax control requirements and improve performance in assembly and insertion tasks. Previous work has focused on special end effector solutions for isolated tasks, such as the peg-in-hole task. However, as many robot tasks require the precision of rigid robots, and their performance would degrade when simply adding compliance, it has been difficult to take advantage of physical softness in real applications. A wrist that could switch between soft and rigid modes could solve this problem, but actuators with sufficient strength for this state transition would increase the size and weight of the module and decrease the payload of the robot. To solve this problem, we propose a novel design of a soft module consisting of a cable-driven mechanism, which allows the robot end effector to change between soft and rigid mode while being very compact and light. The module effectively combines the advantages of soft and rigid robots, and can be retrofitted to existing robots and grippers while preserving the characteristics of the robotic system. We evaluate the effectiveness of our proposed design through experiments modeling assembly tasks, and investigate design parameters quantitatively.

Physically soft robots are promising for robotic assembly tasks as they allow stable contacts with the environment. In this study, we propose a novel learning system for soft robotic assembly strategies. We formulate this problem as a reinforcement learning task and design the reward function from human demonstrations. Our key insight is that the failed demonstrations can be used as constraints to avoid failed behaviors. To this end, we developed a teaching device with which humans can intuitively provide various demonstrations. Moreover, we leverage Physically-Consistent Gaussian Mixture Models to clearly assign Gaussian components to the successful and failed trials. We then create the reference trajectories via Gaussian Mixture Regressions, which fit the successful demonstrations while considering the failed ones. Finally, we apply a sample- efficient deep model-based reinforcement learning method to obtain robust strategies with a few interactions. To validate our method, we developed a real-robot experimental system composed of a rigid collaborative robot arm with a compliant wrist and the teaching device. Our results demonstrated that our method learned the assembly strategies with a higher success rate than when using only successful demonstrations.

We describe experiments towards building a conversational digital assistant that considers the preferred conversational style of the user. In particular, these experiments are designed to measure whether users prefer and trust an assistant whose conversational style matches their own. To this end we conducted a user study where subjects interacted with a digital assistant that responded in a way that either matched their conversational style, or did not. Using self-reported personality attributes and subjects' feedback on the interactions, we built models that can reliably predict a user's preferred conversational style.

Computer (digital) forensic examiners typically write a report to document the examination process, including tools used, major processing steps, summary of the findings, and a detailed listing of relevant evidence (files, artifacts) exported to external media (CD, DVD, hard copy) for the case investigator or attorney. However, proper interpretation of the significance of extracted evidence often requires additional consultation with the examiner. This paper proposes a practical methodology for transforming the findings in typical forensic reports to a graphical representation using Bayesian networks (BNs). BNs offer the following advantages: (1) Delineate the cause-effect relationship among relevant pieces of evidence described in the report; and (2) Use probability and established Bayesian inference rules to deal with uncertainty of digital evidence. A realistic forensic report is used to demonstrate this methodology.

In this paper, we present a framework for simulating light transport in three-dimensional tissue with inhomogeneous scattering properties. Our approach employs a computational model to simulate light scattering in tissue through the finite element solution of the diffusion equation. Although our model handles both visible and nonvisible wavelengths, we especially focus on the interaction of near infrared (NIR) light with tissue. Since most human tissue is permeable to NIR light, tools to noninvasively image tumors, blood vasculature, and monitor blood oxygenation levels are being constructed. We apply this model to a numerical phantom to visually reproduce the images generated by these real-world tools. Therefore, in addition to enabling inverse design of detector instruments, our computational tools produce physically-accurate visualizations of subsurface structures.

A tension exists between designers' comfort with physical artifacts and the need for effective remote collaboration: physical objects live in one place. Previous research and technologies to support remote collaboration have focused on shared electronic media. Current technologies force distributed teams to choose between the physical tools they prefer and the electronic communication mechanisms available. We present Distributed Designers' Outpost, a remote collaboration system based on The Designers' Outpost, a collaborative web site design tool that employs physical Post-it notes as interaction primitives. We extended the system for synchronous remote collaboration and introduced two awareness mechanisms: transient ink input for gestures and a blue shadow of the remote collaborator for presence. We informally evaluated this system with six professional designers. Designers were excited by the prospect of physical remote collaboration but found some coordination challenges in the interaction with shared artifacts.

To form a deep understanding of the present; we need to ?nd and engage history. We present an informal history capture and retrieval mechanism for collaborative, early-stage information design. This history system is implemented in the context of the Designers' Outpost, a wall-scale, tangible interface for collaborative web site design. The interface elements in this history system are designed to be ?uid and comfortable for early-phase design. As demonstrated by an informal lab study with six professional designers, this history system enhances the design process itself, and provides new opportunities for reasoning about the design of complex artifacts

The detection and classification of unexploded ordnances (UXOs) is a difficult task, and it is even further complicated by the fact that the ground penetrating radar (GPR) antenna can significantly distort the scattering signal from the UXO. We consider the model of the real antenna which will be used in the numerical study of scattering from UXOs. We consider a rigorous finite difference time domain (FDTD) model of a fully polarimetric horn-fed bowtie (HFB) antenna and study how various modifications of the parameters of the antenna can affect it performance.

This paper presents a new static type system for multithreaded programs; well-typed programs in our system are guaranteed to be free of data races and deadlocks. Our type system allows programmers to partition the locks into a fixed number of equivalence classes and specify a partial order among the equivalence classes. The type checker then statically verifies that whenever a thread holds more than one lock, the thread acquires the locks in the descending order.Our system also allows programmers to use recursive tree-based data structures to describe the partial order. For example, programmers can specify that nodes in a tree must be locked in the tree order. Our system allows mutations to the data structure that change the partial order at runtime. The type checker statically verifies that the mutations do not introduce cycles in the partial order, and that the changing of the partial order does not lead to deadlocks. We do not know of any other sound static system for preventing deadlocks that allows changes to the partial order at runtime.Our system uses a variant of ownership types to prevent data races and deadlocks. Ownership types provide a statically enforceable way of specifying object encapsulation. Ownership types are useful for preventing data races and deadlocks because the lock that protects an object can also protect its encapsulated objects. This paper describes how to use our type system to statically enforce object encapsulation as well as prevent data races and deadlocks. The paper also contains a detailed discussion of different ownership type systems and the encapsulation guarantees they provide.

A numerical model for scattering from sea ice based on the finite difference time domain (FDTD) technique is presented. The sea ice medium is modeled as consisting of randomly located spherical brine scatterers with a specified fractional volume, and the medium is modeled both with and without a randomly rough boundary to study the relative effects of volume and surface scattering. A Monte Carlo simulation is used to obtain numerical results for incoherent /spl upsi//spl upsi/ backscattered normalized radar cross sections (RCSs) in the frequency range from 3 to 9 GHz and for incidence angles from 10/spl deg/ to 50/spl deg/ from normal incidence. The computational intensity of the study necessitates an effective permittivity approach to modeling brine pocket effects and a nonuniform grid for small scale surface roughness. However, comparisons with analytical models show that these approximations should introduce errors no larger than approximately 3 dB. Incoherent /spl upsi//spl upsi/ cross sections backscattered from sea ice models with a smooth surface show only a small dependence on incidence angle, while results for sea ice models with slightly rough surfaces are found to be dominated by surface scattering at incidence angles less than 30/spl deg/ and by scattering from brine pockets at angles greater than 30/spl deg/. As the surface roughness increases, surface scattering tends to dominate at all incidence angles. Initial comparisons with measurements taken with artificially grown sea ice are made, and even the simplified sea ice model used in the FDTD simulation is found to provide reasonable agreement with measured data trends. The numerical model developed ran be useful in interpreting measurements when parameters such as surface roughness and scatterer distributions lie outside ranges where analytical models are valid.

OBJECTIVE: To evaluate the performance of tools for authoring patient database queries. DESIGN: Query by Review, a tool that exploits the training that users have undergone to master a result review system, was compared with AccessMed, a vocabulary browser that supports lexical matching and the traversal of hierarchical and semantic links. Seven subjects (Medical Logic Module authors) were asked to use both tools to gather the vocabulary terms necessary to perform each of eight laboratory queries. MEASUREMENTS: The proportion of queries that were correct; intersubject agreement. RESULTS: Query by Review had better performance than AccessMed (38% correct queries versus 18%, p = 0.002), but both figures were low. Poor intersubject agreement (28% for Query by Review and 21% for AccessMed) corroborated the relatively low performance. Subjects appeared to have trouble distinguishing laboratory tests from laboratory batteries, picking terms relevant to the particular data type required, and using classes in the vocabulary's hierarchy. CONCLUSION: Query by Review, with its more constrained user interface, performed somewhat better than AccessMed, a more general tool. Neither tool achieved adequate performance, however, which points to the difficulty of formulating a query for a clinical database and the need for further work.