Image inpainting is the task of filling in missing or masked regions of an image with semantically meaningful content. Recent methods have shown significant improvement in dealing with large missing regions. However, these methods usually require large training datasets to achieve satisfactory results, and there has been limited research into training such models on a small number of samples. To address this, we present a novel data-efficient generative residual image inpainting method that produces high-quality inpainting results. The core idea is to use an iterative residual reasoning method that incorporates convolutional neural networks (CNNs) for feature extraction and transformers for global reasoning within generative adversarial networks, along with image-level and patch-level discriminators. We also propose a novel forged-patch adversarial training strategy to create faithful textures and detailed appearances. Extensive evaluation shows that our method outperforms previous methods on the data-efficient image inpainting task, both quantitatively and qualitatively.

Existing facial editing methods have achieved remarkable results, yet they often fall short in supporting multimodal conditional local facial editing. One of the significant evidences is that their output image quality degrades dramatically after several iterations of incremental editing, as they do not support local editing. In this paper, we present a novel multimodal generative and fusion framework for globally-consistent local facial editing (FACEMUG) that can handle a wide range of input modalities and enable fine-grained and semantic manipulation while remaining unedited parts unchanged. Different modalities, including sketches, semantic maps, color maps, exemplar images, text, and attribute labels, are adept at conveying diverse conditioning details, and their combined synergy can provide more explicit guidance for the editing process. We thus integrate all modalities into a unified generative latent space to enable multimodal local facial edits. Specifically, a novel multimodal feature fusion mechanism is proposed by utilizing multimodal aggregation and style fusion blocks to fuse facial priors and multimodalities in both latent and feature spaces. We further introduce a novel self-supervised latent warping algorithm to rectify misaligned facial features, efficiently transferring the pose of the edited image to the given latent codes. We evaluate our FACEMUG through extensive experiments and comparisons to state-of-the-art (SOTA) methods. The results demonstrate the superiority of FACEMUG in terms of editing quality, flexibility, and semantic control, making it a promising solution for a wide range of local facial editing tasks.

Traditional baseline-subtraction methods in calculating pupillary response have short-comings in assessing workloads. The enhanced algorithms based on pupil oscillation, such as the Index of Cognitive Activity (ICA), Index of Pupillary Activity (IPA), and the enhanced version, Low/High Index of Pupillary Activity (LHIPA), utilizing wavelet analysis, have been developed to measure cognitive loads in scenarios where the gaze is relatively static. The applicability of these algorithms in motor tasks, where eye-hand coordination plays a significant role, has not been sufficiently studied. This research explores the effectiveness of LHIPA in measuring task workloads of users in a simulated surgical task. The task’s complexity was determined using Fitts’ index of difficulty, which varied by target size and distance. We found that LHIPA effectively differentiates task workloads across various levels of task difficulties in the surgical task. Results are crucial for creating objective methods to assess task workloads using pupil-based metrics during goal-oriented movements.

Research has found that changes in mental workload during both cognitive and motor tasks can be indicated by pupil dilation. To measure cognitive workload, researchers have developed tools such as the wavelet analysis-based index of cognitive activity (ICA) and index of pupil activity (IPA). However, it is still unclear whether these tools can accurately measure workload during motor tasks. This study aims to investigate whether the IPA can differentiate task workload during a motor task involving aiming in a tele-operation setting, where the task requirements were quantified using Fitts’ index of difficulty(ID) based on target size and distance. The study found that the IPA can differentiate between different levels of motor task workload, provided the proper window is used, namely the period before the tool touches the targets. This finding is significant as it can aid in developing objective methods to evaluate task workload using pupil parameters during goal-directed movements.

Blinks have been widely studied in various fields including medical and human computer interactions, and in driver fatigue. Automatic detection of blinks has valuable practical importance. While existing deep neural networks excel in extracting spatial features from images and demonstrate impressive performance in visual object recognition, their application for blink detection in videos on a frame-by-frame basis is suboptimal, as they only consider spatial features from single images. In this paper, we developed a spatial-temporal-aware deep learning framework that capitalizes on the rapid advancements of the existing state-of-the-art visual object recognition networks, aiming to enhance their performance specifically in blink detection. Our framework takes consecutive frames as input to extract spatial and temporal features simultaneously for better detection of eye movements. We also propose a sliding window re-sampling strategy to mitigate overfitting on training data. Extensive experimental evaluations and comparisons demonstrate the feasibility of the proposed algorithm, which delivers excellent performance for detecting blinks.

We examined whether human operators move their eyes earlier to a target before hands when the level of task difficulty increases. We hypothesized that participants would perform less proactive eye movements in the difficult task than in the easy one, as they would need to focus more on their current hand movements. Sixteen university students were recruited to perform the aiming and touching task reciprocally on three paired targets (circles) differing in sizes and distances, while had their eye movements tracked. The movement time, the early eye engagement time (EEET), and the number of eye adjustments were recorded. The EEET was defined as the time that a subject’s eyes fix on a target to the moment that the tool reaches out for it. The movement time increased as the index of difficulty (ID) value increased, echoing with the Fitts’ Law prediction. When aiming to a target with a higher ID, participants’ EEET was longer comparing to when reaching for a target with a lower ID. Participants reduced the movement speed to give themselves a longer time in searching visually for the target information before moving their hands. In contrast to our hypothesis, results suggested a proportional relationship between the task difficulty and the early eye engagement time. Participants also performed an increasing number of eye adjustments over the course of moving from the easy to the hard target. Future research is needed to examine eye hand coordination under the regulation of Fitts’ Law.

This work studies the influence of slice permutations on tensor recovery, which is derived from a reasonable assumption about algorithm, i.e. changing data order should not affect the effectiveness of the algorithm. However, as we will discussed in this paper, this assumption is not satisfied by tensor recovery under some cases. We call this interesting problem as Slice Permutations Variability (SPV) in tensor recovery. In this paper, we discuss SPV of several key tensor recovery problems theoretically and experimentally. The obtained results show that there is a huge gap between results by tensor recovery using tensor with different slices sequences. To overcome SPV in tensor recovery, we develop a novel tensor recovery algorithm by Minimum Hamiltonian Circle for SPV (TRSPV) which exploits a low dimensional subspace structures within data tensor more exactly. To the best of our knowledge, this is the first work to discuss and effectively solve the SPV problem in tensor recovery. The experimental results demonstrate the effectiveness of the proposed algorithm in eliminating SPV in tensor recovery.

This paper describes a study that explores the force exertion effect on the classification of grasps using a force myography (FMG) technology. Nine participants were recruited to the study; each performed a set of 16 different grasps from a grasp taxonomy using eight different levels of force, respectively. Their wrist muscle pressure was recorded using an array of 16 force sensing resistors. A linear discriminant analysis model was trained by grasps at a single force level using the natural grasping force to classify grasps generated by eight different levels of force. The results show that the grasping force significantly affects the accuracy of grasp classification such that a grasping force closer to the natural force achieves a higher accuracy. A still acceptable classification performance can be achieved for approximately half of the natural grasping force. The findings of this study help the understanding of how force exertion can affect grasp recognition using FMG. Knowledge gained from this study will provide guidance for the implementation of gesture control interfaces in terms of grasping force variations.

Step counting is a practical way for evaluating the activity level of people in daily life. However, the widely used accelerometer-based step-counters are not able to accurately detect low-speed steps (<;0.6 m/s). Our earlier study used supervised machine learning to achieved a very high performance (error rate <;1.5%) in low speed step detection based on the force myography (FMG) signals recorded at the ankle. However, the supervised machine learning approach requires a training process using carefully labelled data. The present study explores the feasibility of using unsupervised learning technique to improve the usability of the ankle force sensing resisters (FSR) band in step detection. An unsupervised K-Means algorithm was employed to train and test with the FMG data recorded from an array of 8 FSRs worn on the ankle position. Eight young healthy volunteers participated in the study by walking on a treadmill at 3 different speeds (0.28 m/s, 0.42 m/s, and 0.56 m/s) while FMG signals were recorded. Results showed a low error rate in the step detection (2.2%) at all 3 walking speeds using the unlabelled data for training.

Mobile and wearable devices based activity recognition systems utilize built-in sensors to identify the activities performed by users pervasively. However, most of these systems do not explicitly present the sensing process to users and are prone to uncertainty. The presence of uncertainty makes users feel confused about the behaviors of activity recognition systems, which may affect the confidence of users. Uncertainty visualization has become an interesting research topic purporting to help users better understand systems. In this paper, we present an uncertainty visualization to reveal the process of mobile and wearable devices based activity recognition systems. We conducted an experiment to evaluate the uncertainty visualization by using a particular simulated mobile and wearable devices based activity recognition application. The results showed that the uncertainty visualization was effective in helping users understand and trust the judgments and inferences of the activity recognition application. Based on the advice of participants, we concluded a few directions to improve the uncertainty visualization.

Pupil size is known to correlate with the changes of cognitive task workloads, but how the pupil responds to requirements of basic goal-directed motor tasks involved in human-machine interactions is not yet clear. This work conducted a user study to investigate the pupil dilations during aiming in a tele-operation setting, with the purpose of better understanding how the changes in task requirements are reflected by the changes of pupil size. The task requirements, managed by Fitts' index of difficulty (ID), i.e. the size and distance apart of the targets, were varied between tasks, and pupil responses to different task IDs were recorded. The results showed that pupil diameter can be employed as an indicator of task requirements in goal-directed movements-higher task difficulty evoked higher valley to peak pupil dilation, and the peak pupil dilation occurred after a longer delay. These findings contribute to the foundation for developing methods to objectively evaluate interactive task requirements using pupil parameters during goal-directed movements in HCI.

Pupil size is known to correlate with changes of cognitive task workloads, but the pupillary response to requirements of basic goal-directed motor tasks is not yet clear, although pointing with tools is a ubiquitous human task. This work describes a user study to investigate the pupil dilations during aiming in two tele-operation tasks with different target settings, one aiming at targets with different sizes located at constant distance apart, and the other aiming at targets varying in different distances. The task requirements in each task were defined by Fitts' index of difficulty (ID). The purpose of this work is to further explore how the changes in task requirements are reflected by the changes of pupil size, i.e., whether the pupil responds to either target size or target distance, or to both of them. Pupil responses to different task IDs were recorded in each task. The results showed that the pupil responds to the changes of ID, not just to the change of target size. This implies that pupil diameter can be employed as an indicator of task requirement in goal-directed movements, because higher task difficulty evoked higher peak pupil dilation which occurred with longer delay. These findings can be used for detailed understanding of eye-hand coordination mechanisms in interactive systems and contribute to the foundation for developing methods to objectively evaluate interactive task requirements using pupil parameters during goal-directed movements.

Novices were trained to perform a unimanual peg transport task in a laparoscopic training box with an illuminated interior displayed on a monitor. Subjects were divided into two groups; one group was verbally instructed to direct their gaze at distant targets, while the other group had their gaze behaviour implicitly manipulated using distant target illumination. Both groups achieved similar task completion times post-training and developed peripheral vision strategies leading to delayed foveation on targets until the instrument was closer to its destination, although the ability to focus on targets earlier during manual movements as done by an expert surgeon was quickly regained by the verbal instruction group post-training. This suggests that care should be taken when employing visual attention cuing methods such as target highlighting for training eye-hand coordination skills, as simple verbal instruction may be sufficient to help trainees to adopt more expert-like gaze behaviours.

To observe whether there is a difference in eye gaze between doing a task, and watching a video of the task, we recorded the gaze of 17 subjects performing a simple surgical eye-hand coordination task. We also recorded eye gaze of the same subjects later while they were watching videos of their performance.