As technological advancements continue to expand the capabilities of multi unmanned-aerial-vehicle systems (mUAV), human operators face challenges in scalability and efficiency due to the complex cognitive load and operations associated with motion adjustments and team coordination. Such cognitive demands limit the feasible size of mUAV teams and necessitate extensive operator training, impeding broader adoption. This paper developed a Hand Gesture Based Interactive Control (HGIC), a novel interface system that utilize computer vision techniques to intuitively translate hand gestures into modular commands for robot teaming. Through learning control models, these commands enable efficient and scalable mUAV motion control and adjustments. HGIC eliminates the need for specialized hardware and offers two key benefits: 1) Minimal training requirements through natural gestures; and 2) Enhanced scalability and efficiency via adaptable commands. By reducing the cognitive burden on operators, HGIC opens the door for more effective large-scale mUAV applications in complex, dynamic, and uncertain scenarios. HGIC will be open-sourced after the paper being published online for the research community, aiming to drive forward innovations in human-mUAV interactions.

Robotic navigation aids humans in essential scenarios, including airport customer boarding, commercial meeting receptions, and disaster site navigation. While individuals benefit from robotic services, these services pose accessibility challenges for people with disabilities. Robots may face limitations in mobility, vision, hearing, and cognitive reasoning, hindering disabled individuals' access to these services. As robotic services become increasingly prevalent, ensuring they are inclusive and accessible is crucial, accommodating human disabilities with adaptations such as slower movements, standing support, and hazard detection. To address this issue, this study introduces a novel Accessibility-Aware Reinforcement Learning model (ARL). ARL extracts disability-related information from human behavioral observations using a contextembedding neural network. It then adjusts robot motions to provide inclusive assistance to individuals with disabilities. To assess the effectiveness of this approach, four assisting services in navigation scenarios ('lead', 'wait', 'assist', and 'protect') and three application scenarios involving navigating elderly individuals in a museum, through traffic, and across a building patio were designed. The results validate ARL's efficacy in enabling robots to make accessibility-aware decisions, thereby enhancing their support for individuals with disabilities.

Advanced by rich perception and precise execution, robots possess immense potential to provide professional and customized rehabilitation exercises for patients with mobility impairments caused by strokes. Autonomous robotic rehabilitation significantly reduces human workloads in the long and tedious rehabilitation process. However, training a rehabilitation robot is challenging due to the data scarcity issue. This challenge arises from privacy concerns (e.g., the risk of leaking private disease and identity information of patients) during clinical data access and usage. Data from various patients and hospitals cannot be shared for adequate robot training, further compromising rehabilitation safety and limiting implementation scopes. To address this challenge, this work developed a novel federated joint learning (FJL) method to jointly train robots across hospitals. FJL also adopted a long short-term memory network (LSTM)-Transformer learning mechanism to effectively explore the complex tempo-spatial relations among patient mobility conditions and robotic rehabilitation motions. To validate FJL’s effectiveness in training a robot network, a clinic-simulation combined experiment was designed. Real rehabilitation exercise data from 200 patients with stroke diseases (upper limb hemiplegia, Parkinson’s syndrome, and back pain syndrome) were adopted. Inversely driven by clinical data, 300,000 robotic rehabilitation guidances were simulated. FJL proved to be effective in joint rehabilitation learning, performing 20% - 30% better than baseline methods.

Dexterous manipulation in robotics, particularly with high degrees of freedom (DoF) devices like the 24-joint Shadow Hand, confronts complexities in search space and execution precision. Humans, however, manipulate objects effortlessly, thanks to their innate grasp of physics. Inspired by this, we introduce the Physics Representation Learning (PRL) framework for robotic hand manipulation. PRL uses physics principles for action conceptualization, such as aligning axes at specific angles (Figure 1). It is a natural language that describes the physics laws, thus bridging the gap between high-level semantics planning and joint-level execution through inverse kinematics (IK). Built on top of the physics-informed action space, PRL deploys a Reinforcement Learning (RL) network with expert demonstrations. Its effectiveness was validated in Nvidia Isaac Sim on four tasks: hammering, unscrewing, sweeping, and pinching. Results show that PRL significantly outperforms conventional joint-control algorithms and RL without demonstrations, underscoring the benefits of physics-based action representations for complex tasks.

The rapid advancements of drones have demonstrated the versatility and promising potential of human-swarm systems (HSS) across various domains. However, human performance within these systems may be impaired by factors such as limited domain knowledge and mental stress, often leading to cognitive overload and hindering the efficiency and effectiveness of human-swarm teaming. Consequently, the accurate monitoring of cognitive load levels is crucial for optimizing HSS performance. To address the challenges of existing measurement methods, which are often expensive, time-consuming, or lack real-time capabilities, we propose a Large Language Model driven cognitive load prediction framework. This framework integrates comprehensive task context, domain knowledge, and behavior analysis to provide fast and cost-effective predictions in complex scenarios. By leveraging the capabilities of Large Language Models and employing reinforcement learning to model the cognitive load generation, our framework aims to offer real-time insights into human-related factors causing high cognitive load and predict cognitive levels over time, ultimately enhancing the performance of HSS teaming.

Robots assist humans in various activities, from daily living to collaborative manufacturing. Because they have biased learning sources (e.g., data, demonstrations, human feedback), robots inevitably have discriminatory performance regarding individual differences (e.g., skin color, mobility, appearance); discriminatory performance will undermine robots’ service quality, causes request ignorance and response delay, and even cause emotional offenses. Therefore, mitigating biases is critically important for delivering fair robotic services. In this paper, we design a bias-mitigation method – Fairness-Sensitive Policy Gradient Reinforcement Learning (FSPGRL), to help robots self-identify and correct biased behaviors. FSP-GRL identifies bias by examining the abnormal updates along particular gradients and updates the policy network to provide fair decisions. To validate FSPGRL’s effectiveness, we designed a human-centered service scenario: a robot serving people in a restaurant. With a user study involving 24 humans and 1,000 service demonstrations, FSPGRL has proven effective in maintaining fairness during robot services.