The growing presence of service robots in human-centric environments, such as warehouses, demands seamless and intuitive human-robot collaboration. In this paper, we propose a collaborative shelf-picking framework that combines multimodal interaction, physics-based reasoning, and task division for enhanced human-robot teamwork.The framework enables the robot to recognize human pointing gestures, interpret verbal cues and voice commands, and communicate through visual and auditory feedback. Moreover, it is powered by a Large Language Model (LLM) that utilizes Chain of Thought (CoT) and a physics-based simulation engine for safely retrieving cluttered stacks of boxes on shelves, a relationship graph for sub-task generation, extraction sequence planning, and decision making. Furthermore, we validate the framework through real-world shelf-picking experiments such as 1) Gesture-Guided Box Extraction, 2) Collaborative Shelf Clearing, and 3) Collaborative Stability Assistance. This work paves the way for more intuitive and effective human-robot collaboration in warehouse environments. A video demonstrating the real-world implementation of our proposed system is available at: https://youtu.be/353zmxMwESg?si=2_T-D7hUZnzl5522

Object segmentation enhances robotic grasping by aiding object identification. Complex environments and dynamic conditions pose challenges such as occlusion, low light conditions, motion blur and object size variance. To address these challenges, we propose a Bimodal SegNet that fuses two types of visual signals, event-based data and RGB frame data. The proposed Bimodal SegNet network has two distinct encoders — one for RGB signal input and another for Event signal input, in addition to an Atrous Pyramidal Feature Amplification module. Encoders capture and fuse the rich contextual information from different resolutions via a Cross-Domain Contextual Attention layer while the decoder obtains sharp object boundaries. The evaluation of the proposed method undertakes five unique image degradation challenges including occlusion, blur, brightness, trajectory and scale variance on the Event-based Segmentation (ESD) Dataset. The results show a 4%–6% MIOU score improvement over state-of-the-art methods in terms of mean intersection over the union and pixel accuracy. The source code, dataset and model are publicly available at: https://github.com/sanket0707/Bimodal-SegNet.

In near future, robots are envisioned to work alongside humans in unstructured professional and domestic environments. In such setups, collaborative manipulation is a fundamental skill that allows manipulation of heavy loads by load sharing between agents. Grasp planning plays a pivotal role for load sharing but it has not received attention in the literature. This work proposes a grasp analysis approach for collaborative manipulation that allows load sharing by minimizing exerted grasp wrenches in a task specific way. The manipulation task is defined as expected external wrenches acting on the target object. The analysis approach is demonstrated in a two-agent decentralized set-up with unknown objects. After the first agent has grasped the target, the second agent observes the first agent's grasp location and plans its own grasp according to optimal load sharing. The method was verified in a human robot collaborative lifting task. Experiments with multiple objects show that the proposed method results in optimal load sharing despite limited information and partial observability.

Grasping systems that build upon meticulously planned hand postures rely on precise knowledge of object geometry, mass and frictional properties — assumptions which are often violated in practice. In this work, we propose an alternative solution to the problem of grasp acquisition in simple autonomous pick and place scenarios, by utilizing the concept of grasp envelopes: sets of constraints on gripper postures. We propose a fast method for extracting grasp envelopes for objects that fit within a known shape category, placed in an unknown environment. Our approach is based on grasp envelope primitives, which encode knowledge of human grasping strategies. We use environment models, reconstructed from noisy sensor observations, to refine the grasp envelope primitives and extract bounded envelopes of collision-free gripper postures. Also, we evaluate the envelope extraction procedure both in a stand alone fashion, as well as an integrated component of an autonomous picking system.

Grasp planning in multi-robot systems is usually studied in a centralized setting with all robots sharing common knowledge about the overall system. Relaxing this assumption would allow multiple mobile manipulators to cooperate even without strict and precise coordination. Moreover, most typical tasks for cooperative settings, such as transporting heavy objects, require certain forces/torques to be exerted along/around particular directions, for instance, compensating for the weight of the transported object. In this paper, we propose task specific multi-robot grasp planning strategies that allow decentralized planning. Each agent plans its own actions without precise information about the other's plans. The approach is based on analysing a task specific grasp quality metric in a probabilistic context, compensating thus for the incomplete knowledge. Results from simulation experiments demonstrate that task independent planning is clearly inferior when task characteristics are known and thus task specific quality measures should be used. Furthermore, the proposed decentralized planning approaches clearly outperform the baseline and show close to globally optimal performance.

Mobile manipulation of large objects can benefit greatly from the use of multiple cooperating robots. Multi-robot coordination of decentralized systems is, however, challenging due to the nature of such systems. For this reason, many planning problems are yet unexplored. This paper proposes two decentralized approaches for cooperative grasp planning. In our setting, agents do not have information about the embodiments of other agents and each agent thus needs to plan its own actions. The proposed approaches are based on analysing the traditional grasp quality metrics in a probabilistic context which compensates for the incomplete information. Results from simulation experiments demonstrate that the probabilistic approaches clearly improve performance over a baseline approach where the grasps are planned independently. Moreover, their performance is close to that of the optimal centralized approach.

This study presents the design and analysis of an intelligent control system that inherits the systematic and recursive design methodology for an n-link robot manipulator, including actuator dynamics, in order to achieve a high-precision position tracking with a firm stability and robustness. First, the coupled higher order dynamic model of an n-link robot manipulator is introduced briefly. Then, a conventional backstepping control (BSC) scheme is developed for the joint position tracking of the robot manipulator. Moreover, a fuzzy-neural-network-inherited BSC (FNNIBSC) scheme is proposed to relax the requirement of detailed system information to improve the robustness of BSC and to deal with the serious chattering that is caused by the discontinuous function. In the FNNIBSC strategy, the FNN framework is designed to mimic the BSC law, and adaptive tuning algorithms for network parameters are derived in the sense of the projection algorithm and Lyapunov stability theorem to ensure the network convergence as well as stable control performance. Numerical simulations and experimental results of a two-link robot manipulator that are actuated by dc servomotors are provided to justify the claims of the proposed FNNIBSC system, and the superiority of the proposed FNNIBSC scheme is also evaluated by quantitative comparison with previous intelligent control schemes.

This paper presents the design and analysis of an intelligent control system that inherits the robust properties of sliding-mode control (SMC) for an n-link robot manipulator, including actuator dynamics in order to achieve a high-precision position tracking with a firm robustness. First, the coupled higher order dynamic model of an n-link robot manipulator is briefy introduced. Then, a conventional SMC scheme is developed for the joint position tracking of robot manipulators. Moreover, a fuzzy-neural-network inherited SMC (FNNISMC) scheme is proposed to relax the requirement of detailed system information and deal with chattering control efforts in the SMC system. In the FNNISMC strategy, the FNN framework is designed to mimic the SMC law, and adaptive tuning algorithms for network parameters are derived in the sense of projection algorithm and Lyapunov stability theorem to ensure the network convergence as well as stable control performance. Numerical simulations and experimental results of a two-link robot manipulator actuated by DC servo motors are provided to justify the claims of the proposed FNNISMC system, and the superiority of the proposed FNNISMC scheme is also evaluated by quantitative comparison with previous intelligent control schemes.