Force estimation is the core indicator for evaluating the performance of tactile sensors, and it is also the key technical path to achieving precise force feedback mechanisms. This study proposes a design method for a visual tactile sensor (VBTS) that integrates a magnetic perception mechanism, and develops a new tactile sensor called MagicGel. The sensor uses strong magnetic particles as markers and captures magnetic field changes in real time through Hall sensors. On this basis, MagicGel achieves the coordinated optimization of multimodal perception capabilities: it not only has fast response characteristics, but also can perceive non-contact status information of home electronic products. Specifically, MagicGel simultaneously analyzes the visual characteristics of magnetic particles and the multimodal data of changes in magnetic field intensity, ultimately improving force estimation capabilities.

Generative pre-trained models have demonstrated remarkable effectiveness in language and vision domains by learning useful representations. In this paper, we extend the scope of this effectiveness by showing that visual robot manipulation can significantly benefit from large-scale video generative pre-training. We introduce GR-1, a GPT-style model designed for multi-task language-conditioned visual robot manipulation. GR-1 takes as inputs a language instruction, a sequence of observation images, and a sequence of robot states. It predicts robot actions as well as future images in an end-to-end manner. Thanks to a flexible design, GR-1 can be seamlessly finetuned on robot data after pre-trained on a large-scale video dataset. We perform extensive experiments on the challenging CALVIN benchmark and a real robot. On CALVIN benchmark, our method outperforms state-of-the-art baseline methods and improves the success rate from 88.9% to 94.9%. In the setting of zero-shot unseen scene generalization, GR-1 improves the success rate from 53.3% to 85.4%. In real robot experiments, GR-1 also outperforms baseline methods and shows strong potentials in generalization to unseen scenes and objects. We provide inaugural evidence that a unified GPT-style transformer, augmented with large-scale video generative pre-training, exhibits remarkable generalization to multi-task visual robot manipulation. Project page: https://GR1-Manipulation.github.io

In this paper, we present a novel method for mobile manipulators to perform multiple contact-rich manipulation tasks. While learning-based methods have the potential to generate actions in an end-to-end manner, they often suffer from insufficient action accuracy and robustness against noise. On the other hand, classical control-based methods can enhance system robustness, but at the cost of extensive parameter tuning. To address these challenges, we present MOMA-Force, a visual-force imitation method that seamlessly combines representation learning for perception, imitation learning for complex motion generation, and admittance whole-body control for system robustness and controllability. MOMA-Force enables a mobile manipulator to learn multiple complex contact-rich tasks with high success rates and small contact forces. In a real household setting, our method outperforms baseline methods in terms of task success rates. Moreover, our method achieves smaller contact forces and smaller force variances compared to baseline methods without force imitation. Overall, we offer a promising approach for efficient and robust mobile manipulation in the real world. Videos and more details can be found on https://visual-force-imitation.github.io.

The Preisach model and the neural networks are two of the most popular strategies to model hysteresis. In this article, we first mathematically prove that the rate-independent Preisach model is actually a diagonal recurrent neural network (dRNN) with the binary step activation function. For the first time, the hysteresis nature and conditions of the classical dRNN with the tanh activation function are mathematically discovered and investigated, instead of using the common black-box approach and its variants. It is shown that the dRNN neuron is a versatile rate-dependent hysteresis system under specific conditions. The dRNN composed of those neurons can be used for modeling the rate-dependent hysteresis and it can approximate the Preisach model with arbitrary precision with specific parameters for rate-independent hysteresis modeling. Experiments show that the classical dRNN models both kinds of hysteresis more accurately and efficiently than the Preisach model.

In this paper, a compliant five-bar leg mechanism is proposed, designed and manufactured for heavy-load legged robots, by using two magneto-rheological actuators (MRAs) that are capable of offering a maximal torque of 78Nm. To address the rate-dependent hysteresis of the MRA, a hybrid rate-dependent hysteresis model is derived based on the idea of mappings between different hysteresis loops. With integrating the classical Preisach model and the NARX neural network, the hybrid model is able to model hysteresis nonlinearity of the magneto-rheological clutch (MRC). It is then used to estimate and control the output torque of the MRA at the absent of external force/torque sensors. High fidelity force control and variable compliance of the leg mechanism are realized and validated in various experiments with using the MRAs.