Industrial human-robot-collaboration (HRC) depends on under-standing how people act, stay engaged, and work together with robots during shared tasks. In most prior work, these aspects are studied in isolation, which makes it hard to see the full picture of real collaboration. This study tackles that problem with an improved multimodal framework that jointly captures fine-grained human actions, engagement levels, and collaboration outcomes in an industrial assembly setting. It introduces a three-layer analytical model called MICRO-MESO-MACRO (M3), which links detailed action patterns to engagement dynamics and overall system efficiency. All data streams are aligned in time using a custom synchronization process so that visual, motion, and behavioral signals can be analyzed together in a consistent way. The results show clear relationships between varied action patterns, stable engagement, and better collaboration efficiency, providing a solid foundation for designing adaptive, human-aware robotic partners. The annotated dataset and code are available at: https://github.com/arvindsihag/m3_analyzer.

Detecting and interpreting operator actions, engagement, and object interactions in dynamic industrial workflows remains a significant challenge in human-robot collaboration research, especially within complex, real-world environments. Traditional unimodal methods often fall short of capturing the intricacies of these unstructured industrial settings. To address this gap, we present a novel Multimodal Industrial Activity Monitoring (MIAM) dataset that captures realistic assembly and disassembly tasks, facilitating the evaluation of key meta-tasks such as action localization, object interaction, and engagement prediction. The dataset comprises multi-view RGB, depth, and Inertial Measurement Unit (IMU) data collected from 22 sessions, amounting to 290 minutes of untrimmed video, annotated in detail for task performance and operator behavior. Its distinctiveness lies in the integration of multiple data modalities and its emphasis on real-world, untrimmed industrial workflows-key for advancing research in human-robot collaboration and operator monitoring. Additionally, we propose a multimodal network that fuses RGB frames, IMU data, and skeleton sequences to predict engagement levels during industrial tasks. Our approach improves the accuracy of recognizing engagement states, providing a robust solution for monitoring operator performance in dynamic industrial environments. The dataset and code can be accessed from https://github.com/navalkishoremehta95/MIAM/.

Video understanding has become crucial in computer vision research due to the vast amounts of video data generated from various sources. Efficient video analysis, especially the extraction of keyframes from long video sequences, is crucial when combined with state-of-the-art deep learning algorithms for video understanding. This has numerous applications, including smart surveillance, autonomous driving, video summarization, and human activity monitoring in human-robot collaboration (HRC) settings. In the context of Industry 5.0, such analysis aids in enhancing productivity, safety, and ergonomics for industrial workers. However, there is a notable lack of research on adaptive frame selection for video-based Human Activity Recognition (HAR) systems in challenging, uncontrolled industrial environments, leading to high memory and computational demands. To address this, we introduce Dynamic Frame Sampler (DF Sampler), a novel adaptive keyframe sampling method designed to tackle the challenges of processing variable-duration action sequences in complex industrial environments. DF Sampler uses a self-supervised learning approach to adaptively select keyframes, prioritizing segments with significant motion while optimizing the computational efficiency of HAR systems. The method is evaluated on the publicly available HRI30 dataset.

Human-Robot Collaboration (HRC) requires seamless interaction between humans and robots, necessitating an accurate understanding and response to human intentions, despite their unpredictability. This challenge is particularly acute in industrial settings, where the complexity of tasks and the variability of human actions demand robust solutions. Effective Human Activity Recognition (HAR) is essential, particularly where real-time anomaly detection is critical for maintaining productivity and efficiency. To address these challenges, we introduce PROACT, a novel framework that predicts operator actions and detects anomalies by constructing a reference graph from action sequences. By anticipating behavior and identifying deviations early, PROACT enhances safety and operational efficiency. The framework has been validated using the Meccano dataset, which simulates an industrial-like environment, demonstrating its capability in monitoring, proactive guidance, and anomaly detection.

In this paper, we address the intricate challenge of gaze vector prediction, a pivotal task with applications ranging from human-computer interaction to driver monitoring systems. Our innovative approach is designed for the demanding setting of extremely low-light conditions, leveraging a novel temporal event encoding scheme, and a dedicated neural network architecture. The temporal encoding method seamlessly integrates Dynamic Vision Sensor (DVS) events with grayscale guide frames, generating consecutively encoded images for input into our neural network. This unique solution not only captures diverse gaze responses from participants within the active age group but also introduces a curated dataset tailored for low-light conditions. The encoded temporal frames paired with our network showcase impressive spatial localization and reliable gaze direction in their predictions. Achieving a remarkable 100-pixel accuracy of 100%, our research underscores the potency of our neural network to work with temporally consecutive encoded images for precise gaze vector predictions in challenging low-light videos, contributing to the advancement of gaze prediction technologies.