Complex work activity recognition based on wearable sensors is crucial for streamlining work processes in industrial domains. Unlike basic activities such as walking or running, which involve simple repetitive motions, a complex work activity consists of discrete atomic actions such as an action of spreading a shipping label or cutting tape in a packaging task. In addition, the atomic actions sometimes involve characteristic short-term sensor data patterns. In addition, these actions can be performed in different orders by different workers to achieve similar outcomes, resulting in different long-term sensor data trends for different workers. Because multilayer networks for activity recognition may learn short-term features from shallow-level layers and long-term trends from deeper layers, we propose a new transfer learning method called multilevel knowledge transfer (MLKT), which performs level-wise source selection according to trend similarity across workers in different levels. For example, for training shallow layers, highly similar workers are selected for specific short motions (e.g., pasting a shipping label), and to train the deeper layers, workers with similar cadence are selected. This method also enables the adaptive thresholding of source data selection for each layer level during network training using the proposed adaptive level-wise discerning module.

Owing to the cost of collecting labeled sensor data, self-supervised learning (SSL) methods for human activity recognition (HAR) that effectively use unlabeled data for pretraining have attracted attention. However, applying prior SSL to COMPLEX activities in real industrial settings poses challenges. Despite the consistency of work procedures, varying circumstances, such as different sizes of packages and contents in a packing process, introduce significant variability within the same activity class. In this study, we focus on sensor data corresponding to characteristic and necessary actions (sensor data motifs) in a specific activity such as a stretching packing tape action in an assembling a box activity, and propose to train a neural network in self-supervised learning so that it identifies occurrences of the characteristic actions, i.e., Motif Identification Learning (MoIL). The feature extractor in the network is subsequently employed in the downstream activity recognition task, enabling accurate recognition of activities containing these characteristic actions, even with limited labeled training data. The MoIL approach was evaluated on real-world industrial activity data, encompassing the state-of-the-art SSL tasks with an improvement of up to 23.85% under limited training labels.

Unlike human daily activities, existing publicly available sensor datasets for work activity recognition in industrial domains are limited by difficulties in collecting realistic data as close collaboration with industrial sites is required. This also limits research on and development of methods for industrial applications. To address these challenges and contribute to research on machine recognition of work activities in industrial domains, in this study, we introduce a new large-scale dataset for packaging work recognition called OpenPack. OpenPack contains 53.8 hours of multimodal sensor data, including acceleration data, keypoints, depth images, and readings from IoT-enabled devices (e.g., handheld barcode scanners), collected from 16 distinct subjects with different levels of packaging work experience. We apply state-of-the-art human activity recognition techniques to the dataset and provide future directions of complex work activity recognition studies in the pervasive computing community based on the results. We believe that OpenPack will contribute to the sensor-based action/activity recognition community by providing challenging tasks. The OpenPack dataset is available at https://open-pack.github.io.

This study presents a new method for recognizing complex human activities within the logistics domain, such as packaging operations, using acceleration data from a body-worn sensor. The recognition of packaging tasks using standard supervised machine learning is complex because the observed data vary considerably depending on the number of items to be packed, the size of the items, and other parameters. In this study, we focused on the characteristics and necessary key actions (motions) that occur during a specific operation. For instance, when the packaging tape is stretched while assembling the shipping boxes. To focus on these characteristic actions when recognizing data, we propose the use of an attention-based neural network. With our method, the attention-based neural network’s training is guided such that its focus is on the motifs. In addition, this method was designed to accurately recognize short operations by leveraging data augmentation techniques. We tested our method on two logistics datasets and achieved a 3.9% improvement over the previous MGA-Net approach.

This study presents a new method for recognizing complex human activities in a logistical domain, such as packaging, using acceleration data from a body-worn sensor. Recognition of packaging tasks using standard supervised machine learning is difficult because the observed data vary considerably depending on the number of items to pack, the size of the items, and other parameters. In this study, we focus on characteristic and necessary actions (motions) that occur in a specific operation such as an action of stretching packing tape when assembling shipping boxes. We propose the use of an attention-based neural network to focus on these characteristic actions when recognizing the data. However, training of a such deep network model is a data-intensive process, and obtaining a huge amount of labeled training data in actual industrial settings is difficult. To address this problem, we employ motif-detection algorithms to detect sensor data motifs (segments corresponding to characteristic actions) that can be useful for recognizing operations in advance. Moreover, we propose that the training of the attention-based network should be guided such that it pays attention to the detected motifs, i.e., motif-guided training.