Manual material handling tasks, such as lifting and lowering, are ubiquitous across industry sectors. Overexertion during these tasks is among the leading causes of workplace injuries. Previous studies have shown that lifting load is a key factor in determining the risk of injury. However, existing methods for measuring the lifting load often rely on manual measurements, sensor fusion, or other techniques that are difficult to scale in practice. In this study, we present a vision-based approach to automatically predict lifting load by analyzing human body pose trajectories extracted from video alone. Specifically, our method employs person detection, visual tracking, and human body pose estimation to extract pose trajectories and their kinematic features, which are then used to train a Transformer model for load prediction. To evaluate our method, we conducted a human subjects study of 19 participants performing various lifting and lowering tasks with varying postures. Our method achieved an average accuracy of 74.8% to distinguish between light vs. heavy objects, and an average accuracy of 50.8% to identify three levels of lifting loads (light, medium, heavy) across lifting and lowering tasks. These results demonstrate a first step towards computer vision based solutions for automatic, noninvasive, scalable injury risk assessment for manual material handling tasks.

A computer vision (CV) method to automatically measure the revised NIOSH lifting equation asymmetry angle (A) from a single camera is described and tested. A laboratory study involving ten participants performing various lifts was used to estimateAin comparison to ground truth joint coordinates obtained using 3-D motion capture (MoCap). To address challenges, such as obstructed views and limitations in camera placement in real-world scenarios, the CV method utilized video-derived coordinates from a selected set of landmarks. A 2-D pose estimator (HR-Net) detected landmark coordinates in each video frame, and a 3-D algorithm (VideoPose3D) estimated the depth of each 2-D landmark by analyzing its trajectories. The mean absolute precision error for the CV method, compared to MoCap measurements using the same subset of landmarks for estimatingA, was 6.25° (SD = 10.19°, N = 360). The mean absolute accuracy error of the CV method, compared against conventional MoCap landmark markers was 9.45° (SD = 14.01°,N= 360).

Many industrial tasks-such as sanding, installing fasteners, and wire harnessing-are difficult to automate due to task complexity and variability. We instead investigate deploying robots in an assistive role for these tasks, where the robot assumes the physical task burden and the skilled worker provides both the high-level task planning and low-level feedback necessary to effectively complete the task. In this article, we describe the development of a system for flexible human-robot teaming that combines state-of-the-art methods in end-user programming and shared autonomy and its implementation in sanding applications. We demonstrate the use of the system in two types of sanding tasks, situated in aircraft manufacturing, that highlight two potential workflows within the human-robot teaming setup. We conclude by discussing challenges and opportunities in human-robot teaming identified during the development, application, and demonstration of our system.

Collaboration between humans and robots has great promise in manufacturing systems. The utilization of cobots in a manufacturing system can improve both productivity and ergonomics. In this paper, we study the problem of how to allocate limited cobot/robots to manufacturing systems with multiple workstations so that an integrated performance measure, considering both productivity and ergonomics is optimized. Previous work on cobot/robot allocation in manufacturing systems focus on the decomposition of tasks for a single workstation into multiple work elements, and then split them between human and robots, rather than studying multi-machine systems. To bridge this gap, we consider the allocation of cobots/robots to a multi-stage manufacturing system. Specifically, we establish an integrated performance measure and formulate cobot/robot allocation into a constraint integer programming problem. With this formulation, we obtain the optimal allocation of one available cobot/robot in simulated production systems, based on the integrated performance measure of productivity and ergonomics. Furthermore, the allocation problems of production systems with multiple cobots/robots is considered and solved with a scalable algorithm.Note to Practitioners—Collaborative robots and independent robots are increasingly applied to manufacturing production systems. However, how to optimally allocate both types of robots considering both productivity and ergonomics influence has not been well studied. In this article, we established a practical optimization method to allocate cobots/robots to different workstations and split the work between cobot and human in one workstation when there are multiple workstations and a limited number of available cobots/robots in the manufacturing systems. Based on various real-world scenarios, we inferred useful insights for the robot/cobot allocation problem. To deal with the computational load when the number of workstations is large, a scalable optimization algorithm is also adopted. The case study results demonstrated the effectiveness of the proposed approach.

Remotely programming robots to execute tasks often relies on registering objects of interest in the robot's environment. Frequently, these tasks involve articulating objects such as opening or closing a valve. However, existing human-in-the-loop methods for registering objects do not consider articulations and the corresponding impact to the geometry of the object, which can cause the methods to fail. In this work, we present an approach where the registration system attempts to automatically determine the object model, pose, and articulation for user-selected points using nonlinear fitting and the iterative closest point algorithm. When the fitting is incorrect, the operator can iteratively intervene with corrections after which the system will refit the object. We present an implementation of our fitting procedure for one degree-of-freedom (DOF) objects with revolute joints and evaluate it with a user study that shows that it can improve user performance, in measures of time on task and task load, ease of use, and usefulness compared to a manual registration approach. We also present a situated example that integrates our method into an end-to-end system for articulating a remote valve.

Drones can provide a minimally-constrained adapting camera view to support robot telemanipulation. Furthermore, the drone view can be automated to reduce the burden on the operator during teleoperation. However, existing approaches do not focus on two important aspects of using a drone as an automated view provider. The first is how the drone should select from a range of quality viewpoints within the workspace (e.g., opposite sides of an object). The second is how to compensate for unavoidable drone pose uncertainty in determining the viewpoint. In this paper, we provide a nonlinear optimization method that yields effective and adaptive drone viewpoints for telemanipulation with an articulated manipulator. Our first key idea is to use sparse human-in-the-loop input to toggle between multiple automatically-generated drone viewpoints. Our second key idea is to introduce optimization objectives that maintain a view of the manipulator while considering drone uncertainty and the impact on viewpoint occlusion and environment collisions. We provide an instantiation of our drone viewpoint method within a drone-manipulator remote teleoperation system. Finally, we provide an initial validation of our method in tasks where we complete common household and industrial manipulations.

A computer vision method was developed to automatically measure wrist flexion and extension from a 2-D video for occupational health and safety research. Marker-less tracked skeletal joints of the elbow, wrist, and hand estimated the wrist flexion/extension angle between the hand and forearm. Based on the estimated angles, wrist posture was classified as flexion (palmar bending), neutral (no bending), or extension (dorsal bending) for each cycle of hand movement. Applying to a set of laboratory videos of a simulated repetitive motion task, we demonstrated the feasibility of using this algorithm for assessing the state of hand activities during manual work. Tested on 1464 frames from 61 recorded videos for 16 participants, the algorithm achieved an average performance of 72.40% correct, per-class accuracy. The sensitivity and specificity for flexion were 66.16% and 91.47%, respectively. The sensitivity and specificity for extension were 77.12% and 89.72%, respectively. This compared favorably against a previously reported consistency rate of 57% between human analyst estimates and wrist electrogoniometer measured wrist flexion/extension angles. We also applied this technique to 262 video frames of hand flexion instances selected from industrial field video data. For these videos, the average correct per-class accuracy was 76.03% in comparison to human observers. The sensitivity and specificity for flexion were 69.23% and 94.17%, respectively, and the sensitivity and specificity for extension were 91.95% and 80.57%, respectively.

We present situated live programming for human-robot collaboration, an approach that enables users with limited programming experience to program collaborative applications for human-robot interaction. Allowing end users, such as shop floor workers, to program collaborative robots themselves would make it easy to “retask” robots from one process to another, facilitating their adoption by small and medium enterprises. Our approach builds on the paradigm of trigger-action programming (TAP) by allowing end users to create rich interactions through simple trigger-action pairings. It enables end users to iteratively create, edit, and refine a reactive robot program while executing partial programs. This live programming approach enables the user to utilize the task space and objects by incrementally specifying situated trigger-action pairs, substantially lowering the barrier to entry for programming or reprogramming robots for collaboration. We instantiate situated live programming in an authoring system where users can create trigger-action programs by annotating an augmented video feed from the robot’s perspective and assign robot actions to trigger conditions. We evaluated this system in a study where participants (n = 10) developed robot programs for solving collaborative light-manufacturing tasks. Results showed that users with little programming experience were able to program HRC tasks in an interactive fashion and our situated live programming approach further supported individualized strategies and workflows. We conclude by discussing opportunities and limitations of the proposed approach, our system implementation, and our study and discuss a roadmap for expanding this approach to a broader range of tasks and applications.

Objective: This study creates linear and generalized additive models (GAMs) of video-recorded two-dimensional hand motion (synonymously referred to as hand movements or hand kinematics) to predict expert-rated performance along a series of surgical motion scales. Background: Surgical performance assessments are costly and time consuming. Automatically quantifying hand motion may offload some burden of surgical coaching and intervention by automatically collecting features of psychomotor performance. Methods: Five experts rated anonymized video clips of benchtop suturing and tying tasks (n = 219) along four visual-analog (0-10) performance scales: fluidity of motion, motion economy, tissue handling, and hand coordination. Custom software tracked both participant hands across successive video frames and populated a robust feature set to train a series of predictive models to reproduce the expert ratings. Results: A GAM (which accounts for nonlinear effects) predicted fluidity of motion ratings with slope = 0.71, intercept = 1.98, and R2= 0.77 for clinicians of different experience levels. Fluidity of motion and motion economy models outperformed those created to predict hand coordination and tissue handling ratings. Conclusions: Hand motion tracking may not address all contextual features of surgical tasks. Future work will explore how well simulation-based models extrapolate to more dynamic settings of the operating room.

A robust computer vision-based approach is developed to estimate the load asymmetry angle defined in the revised NIOSH lifting equation (RNLE). The angle of asymmetry enables the computation of a recommended weight limit for repetitive lifting operations in a workplace to prevent lower back injuries. An open-source package OpenPose is applied to estimate the 2D locations of skeletal joints of the worker from two synchronous videos. Combining these joint location estimates, a computer vision correspondence and depth estimation method is developed to estimate the 3D coordinates of skeletal joints during lifting. The angle of asymmetry is then deduced from a subset of these 3D positions. Error analysis reveals unreliable angle estimates due to occlusions of upper limbs. A robust angle estimation method that mitigates this challenge is developed. We propose a method to flag unreliable angle estimates based on the average confidence level of 2D joint estimates provided by OpenPose. An optimal threshold is derived that balances the percentage variance reduction of the estimation error and the percentage of angle estimates flagged. Tested with 360 lifting instances in a NIOSH-provided dataset, the standard deviation of angle estimation error is reduced from 10.13° to 4.99°. To realize this error variance reduction, 34% of estimated angles are flagged and require further validation.

Performance-cost trade-offs in video object tracking tasks for long video sequences is investigated. A novel frame-subsampled, drift-resilient (FSDR) video object tracking algorithm is presented that would achieve desired tracking accuracy while dramatically reducing computing time by processing only sub-sampled video frames. A new pattern matching score metric is proposed to estimate the probability of drifting. A drift-recovery procedure is developed to enable the algorithm to recover from a drift situation and resume accurate tracking. Compared against state-of-the-art video object tracking algorithms, dramatic performance (accuracy) enhancement and cost (computing time) reduction are observed.

A novel frame-subsampled, drift-resilient (FSDR) video object tracking (VOT) algorithm is proposed. Two design goals are sought: to improve the accuracy and to reduce the processing time. The drifting problem is mitigated with a drift detector and accompanying drift recovery mechanism. When a drift is detected, the recovery mechanism provides an opportunity to put the tracking back on track. To gather context-dependent statistics required for these procedures, an initial short segment of the video sequence will be used as a training sequence. Thus, this algorithm is more suitable for video sequences much longer than several minutes. To reduce computing time, a novel frame-subsampling strategy is proposed to process the VOT on small subset of frames. The trajectory of the tracked object on frames that are skipped will be estimated via interpolation. Compared with state of art VOT algorithms, dramatic improvement of performance (accuracy) and orders of magnitude computing time reduction are observed.

As collaborative robots begin to appear on factory floors, there is a need to consider how these robots can best help their human partners. In this paper, we propose an optimization framework that generates task assignments and schedules for a human-robot team with the goal of improving both time and ergonomics and demonstrate its use in six real-world manufacturing processes that are currently performed manually. Using the strain index method to quantify human physical stress, we create a set of solutions with assigned priorities on each goal. The resulting schedules provide engineers with insight into selecting the appropriate level of compromise and integrating the robot in a way that best fits the needs of an individual process.

This paper utilizes muscle exertions as a means to affect and study the behavior of participants in a virtual environment. Participants performed a simple lifting task both physically using an actual weight and virtually. In the virtual environment participants were presented with two different types of virtual presentation methods, one in which the weights were shown as a 3D model in the Immersive Visuals scenario and one in which the weights were shown as a simple line in the bland scenario. In the virtual scenarios, the object is only lifted when the participant's muscle activity, measured by surface EMG, exceeds a calibrated minimum level as described in previous literature. We found that while participants were able to perceive the difference for various weights both physically and virtually, we found no significant differences in the perceived efforts between the presentation methods. However, while the participants subjectively indicated that their effort was the same for each of these presentation methods, we found significant differences in the muscle activity between the two virtual presentation methods. For all primary mover muscle groups and weights, the more immersive virtual presentation method led to exertions that were much more approximate to the exertions used for the physical weights.

The perception of objects, depth, and distance has been repeatedly shown to be divergent between virtual and physical environments. We hypothesize that many of these discrepancies stem from incorrect geometric viewing parameters, specifically that physical measurements of eye position are insufficiently precise to provide proper viewing parameters. In this paper, we introduce a perceptual calibration procedure derived from geometric models. While most research has used geometric models to predict perceptual errors, we instead use these models inversely to determine perceptually correct viewing parameters. We study the advantages of these new psychophysically determined viewing parameters compared to the commonly used measured viewing parameters in an experiment with 20 subjects. The perceptually calibrated viewing parameters for the subjects generally produced new virtual eye positions that were wider and deeper than standard practices would estimate. Our study shows that perceptually calibrated viewing parameters can significantly improve depth acuity, distance estimation, and the perception of shape.