The Dynamic Vision Sensor (DVS) is a distinctive visual sensor that exclusively responds to alterations in pixel brightness, enabling the real-time capture of swift and subtle movements with reduced power consumption and data bandwidth requirements. This paper proposes a DVS-aware visual perception method and presents its application for pose estimation of mobile robots. Specifically, a new marker is designed to provide pose reference data that leverages the inherent advantages of DVS more effectively. Moreover, we formulate a pose recognition system incorporating DVS, an algorithm based on Spiking Convolutional Neural Networks (SCNN) and a neuromorphic computing accelerator (Lynxi HS110). Such a formulation can well explore the DVS's advantages, as its event-triggered feature matches the nature of SCNN while the neuromorphic hardware enables efficient, low-power execution, making the system highly suitable for real-time embedded applications. Comparative analysis with traditional ARcode-based pose recognition methods reveals that our innovative approach demonstrates significant advantages in recognition speed and energy efficiency. The whole system is deployed on mobile robots and evaluated in real-world scenarios.

The robotic manipulation of deformable linear objects (DLOs) has broad application prospects in many fields. However, a key issue is to obtain the exact deformation models (i.e., how robot motion affects DLO deformation), which are hard to theoretically calculate and vary among different DLOs. Thus, the shape control of DLOs is challenging, especially for large deformation control that requires global and more accurate models. In this article, we propose a coupled offline and online data-driven method for efficiently learning a global deformation model, allowing for both accurate modeling through offline learning and further updating for new DLOs via online adaptation. Specifically, the model approximated by a neural network is first trained offline on random data, then seamlessly migrated to the online phase, and further updated online during actual manipulation. Several strategies are introduced to improve the model's efficiency and generalization ability. We propose a convex-optimization-based controller and analyze the system's stability using the Lyapunov method. Detailed simulations and real-world experiments demonstrate that our method can efficiently and precisely estimate the deformation model and achieve the large deformation control of untrained DLOs in 2-D and 3-D dual-arm manipulation tasks better than the existing methods. It accomplishes all 24 tasks with different desired shapes on different DLOs in the real world, using only simulation data for the offline learning.

The shape control of deformable linear objects (DLOs) is challenging, since it is difficult to obtain the deformation models. Previous studies often approximate the models in purely offline or online ways. In this paper, we propose a scheme for the shape control of DLOs, where the unknown model is estimated with both offline and online learning. The model is formulated in a local linear format, and approximated by a neural network (NN). First, the NN is trained offline to provide a good initial estimation of the model, which can directly migrate to the online phase. Then, an adaptive controller is proposed to achieve the shape control tasks, in which the NN is further updated online to compensate for any errors in the offline model caused by insufficient training or changes of DLO properties. The simulation and real-world experiments show that the proposed method can precisely and efficiently accomplish the DLO shape control tasks, and adapt well to new and untrained DLOs.