Typing is essential for communication, yet the input behavior of individuals with cerebral palsy (CP) remains underexplored. We investigated 31 CP typists and 31 non-disabled controls using keystroke logging, eye tracking, and motion capture. Our study found that CP typists were slower and less rhythmically stable, but by prioritizing accuracy, their overall keyboard efficiency was comparable to controls. They adopted compensatory visual strategies such as shorter and more frequent fixations, greater reliance on the keyboard, and more gaze shifts, and displayed diverse finger usage strategies from single-finger to multi-finger input. We found that using more fingers did not necessarily result in faster typing. Subtype analysis showed spastic CP typists followed a "slow but steady" rhythm with consistent inter-key intervals, whereas athetoid CP typists exhibited a "fast but unstable" rhythm with greater variability, highlighting distinct mechanisms of typing in CP and providing insights for personalized assistive technologies.

Existing manual labeling of micro-expressions is subject to errors in accuracy, especially in cross-cultural scenarios where deviation in labeling of key frames is more prominent. To address this issue, this paper presents a novel Global Anti-Monotonic Differential Selection Strategy (GAMDSS) architecture for enhancing the effectiveness of spatio-temporal modeling of micro-expressions through keyframe re-selection. Specifically, the method identifies Onset and Apex frames, which are characterized by significant micro-expression variation, from complete micro-expression action sequences via a dynamic frame reselection mechanism. It then uses these to determine Offset frames and construct a rich spatio-temporal dynamic representation. A two-branch structure with shared parameters is then used to efficiently extract spatio-temporal features. Extensive experiments are conducted on seven widely recognized micro-expression datasets. The results demonstrate that GAMDSS effectively reduces subjective errors caused by human factors in multicultural datasets such as SAMM and 4DME. Furthermore, quantitative analyses confirm that offset-frame annotations in multicultural datasets are more uncertain, providing theoretical justification for standardizing micro-expression annotations. These findings directly support our argument for reconsidering the validity and generalizability of dataset annotation paradigms. Notably, this design can be integrated into existing models without increasing the number of parameters, offering a new approach to enhancing micro-expression recognition performance.

Sense of agency refers to the subjective perception of initiating and controlling one’s actions and their consequences within both individual and social contexts. It encompasses not only self-agency, where individuals feel in control of their actions from a first-person perspective, but also other-agency, where individuals comprehend the control exerted by others over their actions through mentalizing. This study explores the characteristics of implicit self-agency and other-agency in social interaction contexts using an interval estimation paradigm. The results indicate that voluntary actions lead to a temporal binding effect in individual contexts. Moreover, the reported intervals generated by the self in individual contexts are shorter than those in human-human and human-robot interaction contexts, suggesting a reduction in self-agency within social interaction contexts. Conversely, temporal intervals in human-human interactions are reported to be shorter than those in individual and human-robot interactions, indicating that other-agency remains unaffected. These findings highlight the distinct effects of social interaction contexts on self-agency and other-agency, offering novel empirical insights into human interaction behavior.

Micro-expressions (MEs) are brief, subtle facial expressions that reveal concealed emotions, offering key behavioral cues for social interaction. Characterized by short duration, low intensity, and spontaneity, MEs have been mostly studied through subjective coding, lacking objective, quantitative indicators. This paper explores ME characteristics using facial electromyography (EMG), analyzing data from 147 macro-expressions (MaEs) and 233 MEs collected from 35 participants. First, regarding external characteristics, we demonstrate that MEs are short in duration and low in intensity. Precisely, we proposed an EMG-based indicator, the percentage of maximum voluntary contraction (MVC%), to measure ME intensity. Moreover, we provided precise interval estimations of ME intensity and duration, with MVC% ranging from 7% to 9.2% and the duration ranging from 307 ms to 327 ms. This research facilitates fine-grained ME quantification. Second, regarding the internal characteristics, we confirm that MEs are less controllable and consciously recognized compared to MaEs, as shown by participants' responses and self-reports. This study provides a theoretical basis for research on ME mechanisms and real-life applications. Third, building on our previous work, we present CASMEMG, the first public ME database including EMG signals, providing a robust foundation for studying micro-expression mechanisms and movement dynamics through physiological signals.

Micro-expressions (MEs) are brief, involuntary facial movements critical for detecting lies, drawing growing interest in psychology and computer science. However, annotating ME can burden human coders with excessive time commitment and overwhelming information that compromises coding reliability and efficiency. Such difficulties in data annotation also led to the small sample size problem and hindered the development of ME analysis. Specifically, our psychological research highlights the complexities involved in human annotation of key frames. To facilitate the annotating process of ME, we proposed the Micro-Expression Key Frame Inference (ME-KFI) problem, aiming to identify MEs’ temporal locations from a single frame, reducing manual annotation effort. We propose a Micro-Expression Contrastive Identification Annotation (MECIA) method as a solution to ME-KFI, including three modules: a contrastive module, an identification module, and an annotation module, corresponding to the three steps of manual annotation. The network’s outputs infer the key frame of ME clips. MECIA demonstrates superior performance over random baselines on SAMM and CAS(ME)$^{2}$databases and maintains comparable recognition accuracy with ground-truth clips.

The purpose of facial expression recognition is to recognize the corresponding emotions. However, people tend to hide their emotions by displaying facial expressions that differ from those evoked by emotions. These inconsistent facial expressions are referred to as masked facial expressions (MFEs). The automatic recognition of hidden emotions within an MFE using image data is challenging. In this study, we find distinctive movement patterns in the facial action units (AUs) of MFE sequences through a detailed analysis. Considering our findings, we propose handcrafted features called dynamic AU intensity features (DAIFs) to represent AU movement. Furthermore, we develop a decoupled AU transformer (DAUT) model for recognition, where the decoupled convolution operators ensure that the temporal information in the DAIF is not damaged. To further improve the recognition performance, we design self-supervised clip prediction for pretraining of DAUT. Experimental results demonstrate that our proposed method performs exceptionally well across all tasks in the MFE dataset, particularly improving accuracy by nearly double on the most challenging 36-class task. This suggests that leveraging temporal information from facial AU movements is a reliable and effective technique for recognizing MFEs.

Micro-expression (ME) is a significant non-verbal communication clue that reveals one person's genuine emotional state. The development of micro-expression analysis (MEA) has just gained attention in the last decade. However, the small sample size problem constrains the use of deep learning on MEA. Besides, ME samples distribute in six different databases, leading to database bias. Moreover, the ME database development is complicated. In this article, we introduce a large-scale spontaneous ME database: CAS(ME)3. The contribution of this article is summarized as follows: (1) CAS(ME)3offers around 80 hours of videos with over 8,000,000 frames, including manually labeled 1,109 MEs and 3,490 macro-expressions. Such a large sample size allows effective MEA method validation while avoiding database bias. (2) Inspired by psychological experiments, CAS(ME)3provides the depth information as an additional modality unprecedentedly, contributing to multi-modal MEA. (3) For the first time, CAS(ME)3elicits ME with high ecological validity using the mock crime paradigm, along with physiological and voice signals, contributing to practical MEA. (4) Besides, CAS(ME)3provides 1,508 unlabeled videos with more than 4,000,000 frames, i.e., a data platform for unsupervised MEA methods. (5) Finally, we demonstrate the effectiveness of depth information by the proposed depth flow algorithm and RGB-D information.

Micro-expression spotting is a fundamental step in the micro-expression analysis. This paper proposes a novel network based convolutional neural network (CNN) for spotting multi-scale spontaneous micro-expression intervals in long videos. We named the network as Micro-Expression Spotting Network (MESNet). It is composed of three modules. The first module is a 2+1D Spatiotemporal Convolutional Network, which uses 2D convolution to extract spatial features and 1D convolution to extract temporal features. The second module is a Clip Proposal Network, which gives some proposed micro-expression clips. The last module is a Classification Regression Network, which classifies the proposed clips to micro-expression or not, and further regresses their temporal boundaries. We also propose a novel evaluation metric for spotting micro-expression. Extensive experiments have been conducted on the two long video datasets: CAS(ME)2and SAMM, and the leave-one-subject-out cross-validation is used to evaluate the spotting performance. Results show that the proposed MESNet effectively enhances the F1-score metric. And comparative results show the proposed MESNet has achieved a good performance, which outperforms other state-of-the-art methods, especially in the SAMM dataset.

Designing a tactile image for blind people is a significant challenge due to the difficulty of recognizing objects on a 2D line drawing image by touch compared to vision. In this paper, we proposed ”tactile accuracy”, a new criterion to evaluate the performance of recognizing 242 raised line images of common objects for 30 subjects (10 blindfolded sighted subjects, 10 congenitally blind subjects, and 10 late blind subjects), instead of the conventional ”naming accuracy” used in the visual image recognition tasks. We used multi-level evaluation criteria including ”tactile accuracy” to systematically analyze the design factors in tactile images. The results showed that using multi-level evaluate criteria could help unveil the tactile cognitive preferences of different types of subjects for personalized learning. Moreover, we reported important design factors that affect tactile image recognition, thus providing guidelines on the design of tactile images.

Deception is a very common phenomenon and its detection can be beneficial to our daily lives. Compared with other deception cues, micro-expression has shown great potential as a promising cue for deception detection. The spotting and recognition of micro-expression from long videos may significantly aid both law enforcement officers and researchers. However, database that contains both micro-expression and macro-expression in long videos is still not publicly available. To facilitate development in this field, we present a new database, Chinese Academy of Sciences Macro-Expressions and Micro-Expressions (CAS(ME)2), which provides both macro-expressions and micro-expressions in two parts (A and B). Part A contains 87 long videos that contain spontaneous macro-expressions and micro-expressions. Part B includes 300 cropped spontaneous macro-expression samples and 57 micro-expression samples. The emotion labels are based on a combination of action units (AUs), self-reported emotion for every facial movement, and the emotion types of emotion-evoking videos. Local Binary Pattern (LBP) was employed for the spotting and recognition of macro-expressions and micro-expressions and the results were reported as a baseline evaluation. The CAS(ME)2database offers both long videos and cropped expression samples, which may aid researchers in developing efficient algorithms for the spotting and recognition of macro-expressions and micro-expressions.

Micro-expressions are brief facial movements characterized by short duration, involuntariness and low intensity. Recognition of spontaneous facial micro-expressions is a great challenge. In this paper, we propose a simple yet effective Main Directional Mean Optical-flow (MDMO) feature for micro-expression recognition. We apply a robust optical flow method on micro-expression video clips and partition the facial area into regions of interest (ROIs) based partially on action units. The MDMO is a ROI-based, normalized statistic feature that considers both local statistic motion information and its spatial location. One of the significant characteristics of MDMO is that its feature dimension is small. The length of a MDMO feature vector is 36 × 2 = 72, where 36 is the number of ROIs. Furthermore, to reduce the influence of noise due to head movements, we propose an optical-flow-driven method to align all frames of a micro-expression video clip. Finally, a SVM classifier with the proposed MDMO feature is adopted for micro-expression recognition. Experimental results on three spontaneous micro-expression databases, namely SMIC, CASME and CASME II, show that the MDMO can achieve better performance than two state-of-the-art baseline features, i.e., LBP-TOP and HOOF.

A majority of video summarization systems use linear representations, such as rectangular storyboards and timelines at linear scales. In this paper, we propose a novel nonlinear dynamic representation called SpiralTape that summarizes a video in a smooth spiral pattern. SpiralTape provides an unusual and fresh activity suitable for stimulating environments such as science and technology museums, in which children or young individuals can have enjoyable experiences that create meaningful learning outcomes. In addition, SpiralTape provides an uninterrupted overall structure of video content and takes design principles including compactness, continuity, efficient overview, and interactivity into consideration. A working SpiralTape system was developed and deployed in pilot applications and exhibitions. Elaborate user studies with evaluation benchmarks on multiple metrics were conducted to compare SpiralTape with two representative linear video summarization methods and a state-of-the-art radial video visualization. The evaluation results demonstrate the effectiveness and natural interaction performance of SpiralTape.

Micro-expressions are brief involuntary facial expressions that reveal genuine emotions and, thus, help detect lies. Because of their many promising applications, they have attracted the attention of researchers from various fields. Recent research reveals that two perceptual color spaces (CIELab and CIELuv) provide useful information for expression recognition. This paper is an extended version of our International Conference on Pattern Recognition paper, in which we propose a novel color space model, tensor independent color space (TICS), to help recognize micro-expressions. In this paper, we further show that CIELab and CIELuv are also helpful in recognizing micro-expressions, and we indicate why these three color spaces achieve better performance. A micro-expression color video clip is treated as a fourth-order tensor, i.e., a four-dimension array. The first two dimensions are the spatial information, the third is the temporal information, and the fourth is the color information. We transform the fourth dimension from RGB into TICS, in which the color components are as independent as possible. The combination of dynamic texture and independent color components achieves a higher accuracy than does that of RGB. In addition, we define a set of regions of interests (ROIs) based on the facial action coding system and calculated the dynamic texture histograms for each ROI. Experiments are conducted on two micro-expression databases, CASME and CASME 2, and the results show that the performances for TICS, CIELab, and CIELuv are better than those for RGB or gray.

Micro-expression is a brief involuntary facial expression which reveals genuine emotions and helps detect lies. It intrigues psychologists and computer scientists' (especially on computer vision and pattern recognition) interests due to its promising applications in various fields. Recent research reveals that color may provide useful information for expression recognition. In this paper, we propose a novel color space model, Tensor Independent Color Space (TICS), for enhancing the performance of micro-expression recognition. An micro-expression color video clip is treated as a fourth-order tensor, i.e. a four-dimension array. The first two dimensions are the spatial information, the third is the temporal information, and the fourth is the color information. We transform the fourth dimension from RGB into TICS, in which the color components are as independent as possible. The combination of dynamic texture in the independent color components can get higher accuracy than that in RGB. In addition, we define a set of Regions of Interest (ROIs) based on Facial Action Coding System (FACS) and calculated the dynamic texture histograms for each ROI. The experiments are conducted on two micro-expression databases, CASME and CASME 2, and the results show that the performance in TICS is better than that in RGB or gray.

As a general framework, Laplacian embedding, based on a pairwise similarity matrix, infers low dimensional representations from high dimensional data. However, it generally suffers from three issues: 1) algorithmic performance is sensitive to the size of neighbors; 2) the algorithm encounters the well known small sample size (SSS) problem; and 3) the algorithm de-emphasizes small distance pairs. To address these issues, here we propose exponential embedding using matrix exponential and provide a general framework for dimensionality reduction. In the framework, the matrix exponential can be roughly interpreted by the random walk over the feature similarity matrix, and thus is more robust. The positive definite property of matrix exponential deals with the SSS problem. The behavior of the decay function of exponential embedding is more significant in emphasizing small distance pairs. Under this framework, we apply matrix exponential to extend many popular Laplacian embedding algorithms, e.g., locality preserving projections, unsupervised discriminant projections, and marginal fisher analysis. Experiments conducted on the synthesized data, UCI, and the Georgia Tech face database show that the proposed new framework can well address the issues mentioned above.

With the rapid growth of video resources, techniques for efficient organization of video clips are becoming appealing in the multimedia domain. In this article, a sketch-based approach is proposed to intuitively organize video clips by: (1) enhancing their narrations using sketch annotations and (2) structurizing the organization process by gesture-based free-form sketching on touch devices. There are two main contributions of this work. The first is a sketch graph, a novel representation for the narrative structure of video clips to facilitate content organization. The second is a method to perform context-aware sketch recommendation scalable to large video collections, enabling common users to easily organize sketch annotations. A prototype system integrating the proposed approach was evaluated on the basis of five different aspects concerning its performance and usability. Two sketch searching experiments showed that the proposed context-aware sketch recommendation outperforms, in terms of accuracy and scalability, two state-of-the-art sketch searching methods. Moreover, a user study showed that the sketch graph is consistently preferred over traditional representations such as keywords and keyframes. The second user study showed that the proposed approach is applicable in those scenarios where the video annotator and organizer were the same person. The third user study showed that, for video content organization, using sketch graph users took on average 1/3 less time than using a mass-market tool Movie Maker and took on average 1/4 less time than using a state-of-the-art sketch alternative. These results demonstrated that the proposed sketch graph approach is a promising video organization tool.

3-D CAD models are an important digital resource in the manufacturing industry. 3-D CAD model retrieval has become a key technology in product lifecycle management enabling the reuse of existing design data. In this paper, we propose a new method to retrieve 3-D CAD models based on 2-D pen-based sketch inputs. Sketching is a common and convenient method for communicating design intent during early stages of product design, e.g., conceptual design. However, converting sketched information into precise 3-D engineering models is cumbersome, and much of this effort can be avoided by reuse of existing data. To achieve this purpose, we present a user-adaptive sketch-based retrieval method in this paper. The contributions of this work are twofold. First, we propose a statistical measure for CAD model retrieval: the measure is based on sketch similarity and accounts for users' drawing habits. Second, for 3-D CAD models in the database, we propose a sketch generation pipeline that represents each 3-D CAD model by a small yet sufficient set of sketches that are perceptually similar to human drawings. User studies and experiments that demonstrate the effectiveness of the proposed method in the design process are presented.

Abstract Content‐aware image retargeting is a technique that can flexibly display images with different aspect ratios and simultaneously preserve salient regions in images. Recently many image retargeting techniques have been proposed. To compare image quality by different retargeting methods fast and reliably, an objective metric simulating the human vision system (HVS) is presented in this paper. Different from traditional objective assessment methods that work in bottom‐up manner (i.e., assembling pixel‐level features in a local‐to‐global way), in this paper we propose to use a reverse order (top‐down manner) that organizes image features from global to local viewpoints, leading to a new objective assessment metric for retargeted images. A scale‐space matching method is designed to facilitate extraction of global geometric structures from retargeted images. By traversing the scale space from coarse to fine levels, local pixel correspondence is also established. The objective assessment metric is then based on both global geometric structures and local pixel correspondence. To evaluate color images, CIE L*a*b* color space is utilized. Experimental results are obtained to measure the performance of objective assessments with the proposed metric. The results show good consistency between the proposed objective metric and subjective assessment by human observers.

In the field of image quality assessment, to develop computerized/objective methods whose evaluations are in close agreement with human judgments becomes a main task. However, accurate evaluation of human's subjective judgments is still a problem. Tradition methods based on mean opinion score (MOS) were not accurate enough, especially for images of minor changes or distortions. The present study tried to apply signal detection theory (SDT) in the field of image quality assessment, since SDT is particularly useful in measuring the way we make decisions under conditions of uncertainty. The results of three psychophysics experiments, in which images of different watermarking strengths were used as stimuli, showed that the SDT-based method was especially useful to detect the small loss of fidelity of images. This conclusion was supported by the higher correlation between the sensitivity score, P(A), with several computerized/objective QA indexes, such as PSNR, VIF and SSIM. Detecting subtle changes of images might involve some unknown implicit mechanisms for participants did not perform well enough in full-reference framework which allowing direct comparisons of the changed image to the original one.

The effects of gaze direction on the perception of facial expressions have been getting more and more attention from psychologists, but their implication for facial expression recognition in computer science is neglected. In the present psychological study, we investigated the effects of gaze direction on the perception of facial expressions, and focused on the role of social dominance of facial expressions. Experiments 1 and 2 examined the effects of gaze on evaluations of dominance, pleasure and arousal of neutral, joy, angry and disgust facial expressions, using Self-Assessment Manikin (SAM). Experiment 3 examined the influence of gaze direction on the efficiency of facial expression perception. The results indicated that (a) gaze direction influenced the dominance of neutral and angry facial expressions, while it did not influence the dominance of happy or disgust facial expressions; and (b) averted gaze facilitated the processing of angry faces. The results of three experiments suggested that for threatening facial expression (such as anger), social dominance might be the mediator in the effect of gaze on perception of facial expressions. The implication for facial expression recognition of computer science was discussed.

The information transformation problem of model driven software development (MDSD) processes is gaining increasing attentions. In this paper, we present an indepth analysis of the major causes leading to the ambiguities of existing transformation. A metamodel based approach is then proposed to address these causes. A mechanism to generate transformation rules is devised based on a classification of the UML metamodel elements to be used in subsequent transformation processes. Some other critical issues relating to the generated transformation system are also discussed.

More and more organizations are forming teams that are not co-located. These teams communicate via email, fax, telephone and audio conferences, and sometimes video. The question often arises whether the cost of video is worth it. Previous research has shown that video makes people more satisfied with the work, but it doesnt help the quality of the work itself. There is one exception; negotiation tasks are measurably better with video. In this study, we show that the same effect holds for a more subtle form of negotiation, when people have to negotiate meaning in a conversation. We compared the performance and communication of people explaining a map route to each other. Half the pairs have video and audio connections, half only audio. Half of the pairs were native speakers of English; the other half were non-native speakers, those presumably who have to negotiate meaning more. The results showed that non-native speaker pairs did benefit from the video; native speakers did not. Detailed analysis of the conversational strategies showed that with video, the non-native speaker pairs spent proportionately more effort negotiating common ground.