Diffusion generative modeling has become a promising approach for learning robotic manipulation tasks from stochastic human demonstrations. In this paper, we present Diffusion-EDFs, a novel SE(3)-equivariant diffusion-based approach for visual robotic manipulation tasks. We show that our proposed method achieves remarkable data efficiency, requiring only 5 to 10 human demonstrations for effective end-to-end training in less than an hour. Furthermore, our benchmark experiments demonstrate that our approach has superior generalizability and robustness compared to state-of-the-art methods. Lastly, we validate our methods with real hardware experiments. Project Website: https://sites.google.com/view/diffusion-edfs

In this paper, we introduce a novel approach for designing invariant control laws through potential functions for fully actuated dynamical systems evolving on manifolds by leveraging the power of neural networks. The geometry and non-linearity inherent to manifold-based dynamical systems pose challenges for traditional control law design, necessitating techniques with the interplay of differential geometry and dynamical systems for ensuring stability. Apart from stability, performance and optimality are other challenging areas to address for dynamical systems evolving on manifolds. On top of these, the concept of invariance helps us improve learning transferability skills from one scene to another scene. We propose invariant potential functions on manifolds defined by neural networks that can be used to generate elastic forces for asymptotic tracking of trajectories. The weights of the potential function can be tuned to shape the potential functions according to the performance requirements through minimizing a loss function.

Designing a cost function for nonlinear model predictive control (MPC) with a sparse/binary stage cost is challenging. This paper proposes a novel MPC approach with a scheduled quadratic stage cost function that approximates the true stage cost in order to optimally control a nonlinear system with a sparse/binary stage cost. The cost function parameter is optimally scheduled by a parameter scheduling policy obtained by solving a Markov decision process (MDP) constructed from sampled trajectories from any nonlinear MPC solver. The pro-posed approach is implemented into a differential drive wheeled mobile robot (WMR) designed for smart warehousing via the robot operating system (ROS) framework. The simulation and experimental results successfully demonstrate the effectiveness of our MPC approach in cases of the point stabilization problem of a differential drive WMR.

Learning the uncertain dynamical environments for online learning and prediction from noisy sensory measurement streams is essential for various tasks in robotics. Recently, Gaussian process (GP) online learning such as an infinite-horizon Gaussian process (IHGP) has shown effectiveness to cope with non-stationary dynamical random processes in learning hyperparameters online by reducing the computational cost. However, the IHGP was originally proposed to deal with only a single-output. Therefore, to tackle complex real-world problems, we propose a multi-output infinite-horizon Gaussian process (MOIHGP) that generalizes the single-output IHGP to deal with multiple outputs for better prediction. Our approach allows us to consider correlations between multiple outputs for better prediction, even with occlusions in a Bayesian way. Finally, we successfully demonstrate the effectiveness of our approach by benchmark and experimental results. For simulated benchmark experiments with high noise levels, our approach reduced 16.6% of the averaged RMSE value achieved by the single-output IHGP.

With regard to virtual reality (VR) simulators, direct guidance is needed due to the complex procedures and safety of customers compared with simple coin-op amusement games. However, it is difficult for human attendants to repeatedly guide the customer with a high level of service quality. In this article, we propose a new type of robotic attendant for VR simulators, which provides personalized high-quality services and automatically operates the VR simulators in VR theme parks. In addition, the robotic attendant improves customer satisfaction by providing personalized services based on the estimation of the customer's age, gender, and game progress. Finally, we experimentally show that the personalized services provided by the robotic attendant improve customer satisfaction. The results of the satisfaction questionnaire and independent-samples t-test validate our proposed scheme.

This article presents a novel multifidelity-based optimal sampling method to rapidly estimate the shape of an object from the touch-down points on its surface given a highly limited number of sampling trials. The proposed approach attempts to improve the existing shape estimation via tactile exploration, which uses Gaussian process regression for implicit surface modeling with sequential sampling. The main objective is to make the process of sample point selection more efficient and systematic such that the unknown shape can be estimated fast and accurately with highly limited sample points (e.g., less than 1% of the original dataset). Specifically, we propose to select the next best sample point based on two optimization criteria: 1) the mutual information (MI) for uncertainty reduction, and 2) the local curvature for fidelity enhancement. The combination of these two objectives leads to an optimal sampling process that balances between the exploration of the whole shape and the exploitation of the local area where the higher fidelity (or more sampling) is required. Simulation and experimental results successfully demonstrate the advantage of the proposed method in terms of estimation speed and accuracy over the conventional methods. Our approach allows us to reconstruct recognizable three dimensional shapes using only around optimally selected 0.4% of the original dataset.

Wearing head-mounted display (HMD) makes previous research regarding emotion recognition using machine vision ineffective since they utilized entire face images for training. In this paper, we trained the convolutional neural networks (CNNs) which are capable of estimating the emotions from the images of a face wearing a HMD by hiding eyes and eyebrows from existing face-emotion dataset. Our analysis based on the class activation maps show that it is capable of classifying emotions without the eyes and the eyebrows which ar to serve useful information in recognizing emotions. This implies the possibility of estimating the emotions from the images of humans wearing HMDs using machine vision.

In this paper, we propose a driving assistant companion system that provides drivers with useful information using a long short-term memory network in a data-driven fashion. Our system can be viewed as an advanced driver-assistance system for faster learning and better driving. The assistant companion predicts upcoming events on the road by using real-time sensory measurements from range finding sensors and provides informative narrations to enhance learnability and driving performance. In contrast to a conventional navigation system, our system predicts events from the online stream of sensory measurements without resorting to priors and map information. To demonstrate the effectiveness of the proposed system, we implemented our system on The Open Racing Car Simulator and conducted an experimental study with 16 human drivers. Experimental results show that our system enhanced learnability, driving performance, and reliability.

Translating recent advances in abdominal aortic aneurysm (AAA) growth and remodeling (G&R) knowledge into a predictive, patient-specific clinical treatment tool requires a major paradigm shift in computational modeling. The objectives of this study are to develop a prediction framework that first calibrates the physical AAA G&R model using patient-specific serial computed tomography (CT) scan images, predicts the expansion of an AAA in the future, and quantifies the associated uncertainty in the prediction. We adopt a Bayesian calibration method to calibrate parameters in the G&R computational model and predict the magnitude of AAA expansion. The proposed Bayesian approach can take different sources of uncertainty; therefore, it is well suited to achieve our aims in predicting the AAA expansion process as well as in computing the propagated uncertainty. We demonstrate how to achieve the proposed aims by solving the formulated Bayesian calibration problems for cases with the synthetic G&R model output data and real medical patient-specific CT data. We compare and discuss the performance of predictions and computation time under different sampling cases of the model output data and patient data, both of which are simulated by the G&R computation. Furthermore, we apply our Bayesian calibration to real patient-specific serial CT data and validate our prediction. The accuracy and efficiency of the proposed method is promising, which appeals to computational and medical communities.

In this paper, we consider mobile sensor networks that use spatiotemporal Gaussian processes to predict a wide range of spatiotemporal physical phenomena. Nonparametric Gaussian process regression that is based on truncated observations is proposed for mobile sensor networks with limited memory and computational power. We first provide a theoretical foundation of Gaussian process regression with truncated observations. In particular, we demonstrate that prediction using all observations can be well approximated by prediction using truncated observations under certain conditions. Inspired by the analysis, we then propose a centralized navigation strategy for mobile sensor networks to move in order to reduce prediction error variances at points of interest. For the case in which each agent has a limited communication range, we propose a distributed navigation strategy. Particularly, we demonstrate that mobile sensing agents with the distributed navigation strategy produce an emergent, swarming-like, collective behavior for communication connectivity and are coordinated to improve the quality of the collective prediction capability.