Abstract This study explores the integration of information-rich prior knowledge, specifically human gaze data, to enhance representation learning through contrastive methods. We propose gaze-guided contrastive unsupervised representation learning, a novel framework harnessing human gaze data to guide the selection of positive and negative samples for contrastive learning. By leveraging human gaze information, we capture meaningful patterns in visual task dynamics, enabling the agent to acquire effective strategies from demonstrations and achieve superior performance. Our findings demonstrate significant improvements over baseline algorithms, highlighting the value of gaze-guided representation learning in reducing data requirements and accelerating learning. This approach offers broad applicability to vision-based tasks, emphasizing the critical role of human gaze in improving task efficiency and generalization.

This paper presents a novel multi-robot coverage path planning (CPP) algorithm - aka SCoPP - that provides a time-efficient solution, with workload balanced plans for each robot in a multi-robot system, based on their initial states. This algorithm accounts for discontinuities (e.g., no-fly zones) in a specified area of interest, and provides an optimized ordered list of way-points per robot using a discrete, computationally efficient, nearest neighbor path planning algorithm. This algorithm involves five main stages, which include the transformation of the user’s input as a set of vertices in geographical coordinates, discretization, load-balanced partitioning, auctioning of conflict cells in a discretized space, and a path planning procedure. To evaluate the effectiveness of the primary algorithm, a multi-unmanned aerial vehicle (UAV) post-flood assessment application is considered, and the performance of the algorithm is tested on three test maps of varying sizes. Additionally, our method is compared with a state-of-the-art method created by Guasella et al. Further analyses on scalability and computational time of SCoPP are conducted. The results show that SCoPP is superior in terms of mission completion time; its computing time is found to be under 2 mins for a large map covered by a 150-robot team, thereby demonstrating its computationally scalability.

Human-swarm interaction has recently gained attention due to its plethora of new applications in disaster relief, surveillance, rescue, and exploration. However, if the task difficulty increases, the performance of the human operator decreases, thereby decreasing the overall efficacy of the human-swarm team. Thus, it is critical to identify the task difficulty and adaptively allocate the task to the human operator to maintain optimal performance. In this direction, we study the classification of task difficulty in a human-swarm interaction experiment performing a target search mission. The human may control platoons of unmanned aerial vehicles (UAVs) and unmanned ground vehicles (UGVs) to search a partially observable environment during the target search mission. The mission complexity is increased by introducing adversarial teams that humans may only see when the environment is explored. While the human is completing the mission, their brain activity is recorded using an electroencephalogram (EEG), which is used to classify the task difficulty. We have used two different approaches for classification: A feature-based approach using coherence values as input and a deep learning-based approach using raw EEG as input. Both approaches can classify the task difficulty well above the chance. The results showed the importance of the occipital lobe (O1 and O2) coherence feature with the other brain regions. Moreover, we also study individual differences (expert vs. novice) in the classification results. The analysis revealed that the temporal lobe in experts (T4 and T3) is predominant for task difficulty classification compared with novices.

Human-Swarm interaction has attracted a lot of attention for their applications in areas such as exploration, rescue, surveillance, and interplanetary exploration. When humans assume a supervisory or tactician role in managing the robot swarm, the humans' (physiological) state significantly affects the mission performance. In this work, we explore the physiological correlates with the user's tactical decisions in a simulated search and rescue mission. The mission consists of supervising three groups of unmanned aerial vehicles and three groups of unmanned ground vehicles to search for a target building. The mission complexity is increased by introducing static adversarial teams.Due to the adversarial team's presence, the user should employ different tactics to search for a target. While the user interacts with the swarm, brain activity in forms of electroencephalogram (EEG) and eye movements are recorded. 20 participants, with prior experience in playing real-time strategy games, took part in the study. A linear mixed effect model is used to study the correlated physiological features and tactical decisions. Six features are extracted from the physiological data: engagement level, mental workload, Fz-Pz coherence, Fz-O1 coherence, pupil size, and the number of gaze fixations. The results show that mental engagement and Fz-O1 coherence are the important factors in predicting the tactical decisions. Specifically, Fz-O1 coherence in Beta (22.5-30 Hz) and Gamma (38-42 Hz) band is found to be significant.

In physical human-robot interaction, a variable admittance/impedance controller is desired to adjust its controller parameters to enhance the collaboration by minimizing the human effort and maximizing the stability. In this paper, we propose a physiological monitoring approach based on electroencephalogram activities to classify the motor control difficulty and use that information for adjusting an admittance controller. We designed a physical human-robot interaction experiment where the human guides the robot's end-effector across four tasks with varying motor control difficulty. Each task is a combination of high/low damping and fine/gross motor control. During the experiments, we measure the muscle activation information in terms of surface electromyogram from eight channels. Two sets of features based on Riemann geometry and time domain (Hudgins' features) are extracted every 500 ms from the EMG data. A support vector machine classifier is trained on these features to estimate whether the existing admittance parameters are comfortable for the user else an increase/decrease of the damping is suggested. Riemann geometry-based features yielded higher accuracy (85.7%) than the Hudgins' features (69.1%) across 21 participants; however, the performance of these classifiers on the new sessions degraded to 63.1% and 54.5% respectively. To address this issue, we implemented a transfer learning approach using Riemannian features that improved the inter-session detection rate to 73.95%.

The notions of safe and compliant interaction are not sufficient to ensure effective physical human-robot cooperation. To obtain an optimal compliant behavior (e.g., variable impedance/admittance control), assessment techniques are required to measure the effectiveness of the interaction in terms of perceived workload by users. This study investigates electroencephalography (EEG) monitoring as an objective measure to classify workload in cooperative manipulation with compliance. An experimental study is conducted including two types of manipulation (gross and fine) with two admittance levels (low- and high-damping). Performance and self-reported measures indicate that a proper admittance level that enhances perceived workload is task-dependent. This information is used to form a binary classification problem (low- and high-workload) with spectral power density and coherence as the features extracted from EEG data. Using a subject-independent feature selection approach, a subject-dependent Linear Discriminant Analysis (LDA) is used for classification. An average classification rate of 81% is achieved that indicates the reliability of the proposed approach for assessing human workload in interaction with varying compliance across the gross and fine manipulation. Furthermore, to validate our proposed objective measure of workload, we have conducted a second experiment composed of both fine and gross motor tasks. Compared to interaction with a constant admittance, a lower EEG-based workload is observed with an open-loop variable admittance controller. This observation is in agreement with the subjective workload score (NASA-TLX).

This paper presents the design of a novel variable stiffness gripper with two parallel fingers (jaws). Compliance of the system is generated by using permanent magnets as the nonlinear springs. Based on the presented design, the position and stiffness level of the fingers can be adjusted simultaneously by changing the air gap between the magnets. The modeling of magnetic repulsion force and stiffness are presented and verified experimentally. An experiment is also conducted to demonstrate the functionality of the gripper to improve safety when a fragile object was grasped and the gripper collided with an obstacle.