Understanding of bimanual hand-object interaction plays an important role in robotics and virtual reality. However, due to significant occlusions between hands and object as well as the high degree-of-freedom motions, it is challenging to collect and annotate a high-quality, large-scale dataset, which prevents further improvement of bimanual hand-object interaction-related baselines. In this work, we propose a new 3D Gaussian Splatting based data augmentation framework for bimanual hand-object interaction, which is capable of augmenting existing dataset to large-scale photorealistic data with various hand-object pose and viewpoints. First, we use mesh-based 3DGS to model objects and hands, and to deal with the rendering blur problem due to multi-resolution input images used, we design a super-resolution module. Second, we extend the single hand grasping pose optimization module for the bimanual hand object to generate various poses of bimanual hand-object interaction, which can significantly expand the pose distribution of the dataset. Third, we conduct an analysis for the impact of different aspects of the proposed data augmentation on the understanding of the bimanual hand-object interaction. We perform our data augmentation on two benchmarks, H2O and Arctic, and verify that our method can improve the performance of the baselines.

Object pose estimation, crucial in computer vision and robotics applications, faces challenges with the diversity of unseen categories. We propose a zero-shot method to achieve category-level 6-DOF object pose estimation, which exploits both 2D and 3D universal features of input RGB-D image to establish semantic similarity-based correspondences and can be extended to unseen categories without additional model fine-tuning. Our method begins with combining efficient 2D universal features to find sparse correspondences between intra-category objects and gets initial coarse pose. To handle the correspondence degradation of 2D universal features if the pose deviates much from the target pose, we use an iterative strategy to optimize the pose. Subsequently, to resolve pose ambiguities due to shape differences between intra-category objects, the coarse pose is refined by optimizing with dense alignment constraint of 3D universal features. Our method outperforms previous methods on the REAL275 and Wild6D benchmarks for unseen categories.

Generating high-quality whole-body human object interaction motion sequences is becoming increasingly important in various fields such as animation, VR/AR, and robotics. The main challenge of this task lies in determining the level of involvement of each hand given the complex shapes of objects in different sizes and their different motion trajectories, while ensuring strong grasping realism and guaranteeing the coordination of movement in all body parts. Contrasting with existing work, which either generates human interaction motion sequences without detailed hand grasping poses or only models a static grasping pose, we propose a simple yet effective framework that jointly models the relationship between the body, hands, and the given object motion sequences within a single diffusion model. To guide our network in perceiving the object's spatial position and learning more natural grasping poses, we introduce novel contact-aware losses and incorporate a data-driven, carefully designed guidance. Experimental results demonstrate that our approach outperforms the state-of-the-art method and generates plausible results.

Nowadays, numerous online platforms can be described as multi-modal heterogeneous networks (MMHNs), such as Douban's movie networks and Amazon's product review networks. Accurately categorizing nodes within these networks is crucial for analyzing the corresponding entities, which requires effective representation learning on nodes. However, existing multi-modal fusion methods often adopt either early fusion strategies which may lose the unique characteristics of individual modalities, or late fusion approaches overlooking the cross-modal guidance in GNN-based information propagation. In this paper, we propose a novel model for node classification in MMHNs, named Heterogeneous Graph Neural Network with Inter-Modal Attention (HGNN-IMA). It learns node representations by capturing the mutual influence of multiple modalities during the information propagation process, within the framework of heterogeneous graph transformer. Specifically, a nested inter-modal attention mechanism is integrated into the inter-node attention to achieve adaptive multi-modal fusion, and modality alignment is also taken into account to encourage the propagation among nodes with consistent similarities across all modalities. Moreover, an attention loss is augmented to mitigate the impact of missing modalities. Extensive experiments validate the superiority of the model in the node classification task, providing an innovative view to handle multi-modal data, especially when accompanied with network structures. The full version including Appendix is available at http://arxiv.org/abs/2505.07895.

Recently, rule-based time series classification models are widely used in safety-critical scenarios demanding strong interpretability, through providing explicit and rigorous rationales. However, due to the discrepancy between discrete rules and continuous neural networks, the generated rules are not completely consistent with the actual decision-making process, rendering users hesitant to trust the model. Additionally, existing methods learn the role (weight) of each time point independently, which is not in line with human understanding and the consecutive nature of temporal properties. In this paper, we propose a novel neuro-symbolic model named TemporalRule, aiming to automatically learn Signal Temporal Logic (STL) rules for interpretable time series classification. Our model directly optimizes the neural networks representing discrete rules via gradient grafting, producing reliable rules that can exactly determine the classification results. Notably, a temporal logical layer is designed to simulate expressive temporal operators, including one 2-predicate operator (Until) and two 2-level composite operators (EA and AE), through adaptively learning the time bounds of involved intervals. That creates consistent weights for consecutive time points, enhancing the intuitiveness of derived rules. Extensive experiments on diverse real-world datasets in safety-critical domains show that TemporalRule achieves superior and stable classification accuracy compared to state-of-the-art rule-based approaches, and the learned rules can precisely and concisely explicate the classification results, leading to a trustable model.

The rapid growth of the Internet of Things (IoT) has resulted in an explosive increase in time-series data, making time-series databases (TSDBs) such as InfluxDB and OpenTSDB essential components in IoT ecosystems.At the same time, the decreasing cost of SSDs has facilitated their increasing adoption in large-scale data centers.Traditional TSDBs are primarily based on Log-Structured Merge Tree (LSM-tree) optimized for HDDs, which convert random reads and writes into sequential ones.However, these systems fail to fully exploit the unique characteristics of SSDs, such as random I/O operations and internal parallelism.In this paper, we present ReefsDB, an LSM-tree-based TSDB that is highly optimized for SSDs and implemented using the Rust programming language.We evaluate ReefsDB using the Time Series Benchmark Suite for write-intensive workloads, and the results demonstrate that ReefsDB is 2.9×∼3.2×faster than InfluxDB in write performance, while reducing read latencies by 27%∼62%.

Developing efficient collaborative strategies for UAV swarms is crucial for achieving accurate and rapid execution of the Multi-Target Detecting (MTD) tasks which involve two primary stages in practical scenarios: dispersed search by multi-UAVs in unknown dynamic environments to locate targets, and subsequent aggregation to gather all targets information within the scene, which called detecting and aggregation processes. In recent years, several collaborative strategy methods have been developed for application in UAV swarm mission scenarios. These methods are typically designed for single-stage tasks, and therefore their performance is likely to be suboptimal when applied to multi-stage tasks like the MTD tasks, which have distinctly different characteristics and objectives across stages. In this paper, we propose a novel integrated deep reinforcement learning decision framework that can offer effective collaborative strategies for tasks characterized by distinct stages, denoted as the STDGNet. The STDGNet comprises a Transformer-based Deep Graph Network (TDGN) module alongside two Specialized optimization strategies: the location-dispersion strategy and the cluster-action-consistency strategy. The TDGN module is designed to extract features from observations and interaction dynamics among UAVs, aimed at generating collaborative strategies. The integration of two specialized strategies enables the STDGNet framework to adapt well to multi-stage tasks: during the detection stage, the location-dispersion strategy maintains UAV dispersal to expedite the discovery of more targets; during the aggregation stage, the cluster-action-consistency strategy ensures that UAVs within the same cluster move in the same direction, facilitating the formation of interconnected communication networks. To assess the efficiency and resilience of the proposed framework, we construct an MTD environment where extensive experimentation shows that the STDGNet framework surpasses baseline methodologies.

Online handwriting recognition is pivotal in domains like note-taking, education, healthcare, and office tasks. Existing diagram recognition algorithms mainly rely on the temporal information of strokes, resulting in a decline in recognition performance when dealing with notes that have been modified or have no temporal information. The current datasets are drawn based on templates and cannot reflect the real free-drawing situation. To address these challenges, we present SpaceGTN, a time-agnostic Graph Transformer Network, leveraging spatial integration and removing the need for temporal data. Extensive experiments on multiple datasets have demonstrated that our method consistently outperforms existing methods and achieves state-of-the-art performance. We also propose a pipeline that seamlessly connects offline and online handwritten diagrams. By integrating a stroke restoration technique with SpaceGTN, it enables intelligent editing of previously uneditable offline diagrams at the stroke level. In addition, we have also launched the first online handwritten diagram dataset, OHSD, which is collected using a free-drawing method and comes with modification annotations.

Graphs constructed from real-world scenarios are often incomplete due to privacy restrictions or resource limitations, posing significant challenges for node classification, especially when labeled data are scarce. In many scenarios of incomplete graphs, the real node degrees, such as the number of followers in social networks or publications' references in citation networks, are easily accessible and informative, which could indicate the degree of incompleteness. However, most of existing researches of incomplete graphs focus on edge completion, but ignore the node completion with known node degrees. In this paper, we propose a new few-shot node classification problem on incomplete graphs with real node degrees. To deal with node completion, edge completion and label completion of this problem, we develop an effective Large Language Models (LLMs) empowered Graph Convolutional Network (GCN) model utilizing the real node Degrees, namely LLMDGCN. First, we leverage LLMs to initially fill in the missing nodes and labels. Next, we design an edge prediction module that employs the real node degrees and inter-category probability matrix to recover the missing edges for each node. We then iteratively train the GCN and the edge prediction module. The GCN generates pseudo labels, which the edge prediction module uses to restore edges, and these edges are fed back into the GCN to improve accuracy. Extensive experiments on four benchmark datasets demonstrate the effectiveness and robustness of our proposed method for the few-shot node classification on incomplete graphs with real node degrees.

Imputing missing data in Building Internet of Things (BIoT) poses a significant challenge due to the complex multi-dimensional variations and the inevitable occurrence of long-span missing gaps during maintenance or repair periods. In this paper, we introduce ODE-MI, a novel neural ODE-guided Multiple Imputation framework tailored for BIoT time series with high missing gaps. ODE-MI integrates shapelet-wise convolution, node embeddings, and dynamic graph embeddings with transfer delay to capture compound spatio-temporal (ST) volatilities, trends, and high-order impact transfer. Our contributions include the design of a multi-faceted backbone network capable of capturing long-distance ST characteristics in BIoT, and the development of the ODE-MI framework to handle long-term missing data within networked dynamic attributes. Empirical assessments using open datasets demonstrate that ODE-MI outperforms existing imputation methods, especially in scenarios involving continuous long-span missing gaps.

Deep Neural Networks (DNNs) are crucial for enhancing environmental perception in embedded intelligent systems, particularly in applications such as drones and autonomous vehicles. These mission-critical applications necessitate both computational accuracy and real-time performance. A significant challenge arises from the unpredictable execution times of DNNs and the variable periods of perception tasks. To address this, we present RTDeepEnsemble, an innovative framework designed for the real-time execution of DNN ensembles. RTDeepEnsemble dynamically manages the DNN ensemble as an imprecise computation model, judiciously selecting the optimal DNN for execution at runtime. This framework effectively arranges the DNN execution sequence, estimating utility through a refined Boosting algorithm. Additionally, RTDeepEnsemble prioritizes mandatory subtasks based on the Earliest Deadline First scheduling and employs a dynamic programming method to schedule optional subtasks. Comprehensive evaluations on the Nvidia Jetson Xavier NX platform, within a real-time object detection context, demonstrate RTDeepEnsemble's superiority in both accuracy and deadline miss rate. Notably, while maintaining real-time performance, we observe an accuracy increase of approximately 10 %, underscoring RTDeepEnsemble's potential as a solution for real-time machine perception.

Accurately forecasting traffic flow using time-series data from multimedia sensors remains a significant challenge, despite its importance for advancing intelligent transportation systems. Recent advancements in attention-based models have shown promise in capturing spatial-temporal dependencies in traffic flow data. Yet, these models exhibit three principal limitations: (1) they employ either factorized or coupled spatial-temporal attention mechanisms, potentially failing to fully harness the potential of these distinct approaches; (2) the attention allocation for spatial nodes is predominantly data-centric, which may overlook existing knowledge about the nodes' importance within the transportation network; (3) while traditional attention-based methods effectively capture long-term dependencies, they often struggle with adapting to the disparate lengths of temporal contexts. To overcome these limitations, we introduce a multi-way framework dubbed FEST that innovatively integrates both factorized and coupled spatial-temporal attention mechanisms. We then enhance FEST by incorporating PageRank-derived node importance scores to guide focus on nodes. Moreover, a novel multi-scale temporal learning approach is proposed to improve model capability with both long- and short-term temporal dynamics. Extensive experiments on real-world datasets under long- and short-term prediction scenarios confirm the effectiveness of our method.

A modality gap exists between infrared and visible images, presenting challenges for image fusion. Despite the modality heterogeneity, both types of images inherently capture the same scene, suggesting the presence of common information. Effectively extracting shared features while distinguishing modality-specific ones is pivotal for bridging this gap and achieving superior fusion outcomes. To address this, we propose the S hared-S pecific F usion Net work (S2F-Net). The S2F-Net introduces a three-branch feature extractor, which retains two branches for extracting features from each modality, innovatively creating an additional branch dedicated to facilitating the separation of shared features from modality-specific ones. This facilitates guiding the fusion of cross-modal information to generate efficient fusion features, ensuring the effective integration of complementary information from different modalities. To achieve feature fusion and image reconstruction, we propose two fusion modules: the Cross-modality Attention-Guided Fusion Module (CAGFM) and the Multi-Level Fusion Module (MLFM). The former utilizes shared and specific features by employing cross-modality channel attention, enabling effective integration of information across modalities. The latter facilitates feature interaction across different levels. Additionally, to effectively disentangle shared and specific features, we introduce the shared-specific learning module. Extensive experiments conducted on open-source datasets validate the superior performance of our proposed method.

Weakly supervised text classification (WSTC), also called zero-shot or dataless text classification, has attracted increasing attention due to its applicability in classifying a mass of texts within the dynamic and open Web environment, since it requires only a limited set of seed words (label names) for each category instead of labeled data. With the help of recently popular prompting Pre-trained Language Models (PLMs), many studies leveraged manually crafted and/or automatically identified verbalizers to estimate the likelihood of categories, but they failed to differentiate the effects of these category-indicative words, let alone capture their correlations and realize adaptive adjustments according to the unlabeled corpus. In this paper, in order to let the PLM effectively understand each category, we at first propose a novel form of rule-based knowledge using logical expressions to characterize the meanings of categories. Then, we develop a prompting PLM-based approach named RulePrompt for the WSTC task, consisting of a rule mining module and a rule-enhanced pseudo label generation module, plus a self-supervised fine-tuning module to make the PLM align with this task. Within this framework, the inaccurate pseudo labels assigned to texts and the imprecise logical rules associated with categories mutually enhance each other in an alternative manner. That establishes a self-iterative closed loop of knowledge (rule) acquisition and utilization, with seed words serving as the starting point. Extensive experiments validate the effectiveness and robustness of our approach, which markedly outperforms state-of-the-art weakly supervised methods. What is more, our approach yields interpretable category rules, proving its advantage in disambiguating easily-confused categories.

As video lectures are gaining more popularity, determining their effectiveness and obtaining valuable feedback have become necessary. To measure the learners’ attention state during video lectures, we specified the conceptual “with-me-ness” (WMN) as slide-level WMN (SL-WMN). The content domain on each slide was automatically extracted via an optical character recognition (OCR)-based method, while the eye gazing behaviors were analyzed through a Gaussian mixture modeling (GMM) fixation clustering method. Both domain-specific WMN and behavior-enriched WMN were then computed via OCR- and GMM-OCR-based methods to measure the learners’ attention levels. We conducted an experiment to collect in-lecture eye-tracking data, video recordings, and post-lecture test scores from 50 Grade 8 students. The results demonstrated that both OCR- and GMM-OCR-based SL-WMNs are reliable and compatible automatic measurements of learners’ attention states during video lectures. A survey from participating learners and lecturers also revealed highly favorable feedback for the developed SL-WMNs.

DailyConnect is a visual-based mobile application that supports children with autism spectrum disorder (ASD) in recalling memories by reviewing photos through discrete trial training (DTT) to understand situation-based emotions. To assess DailyConnect and its adaptability to a child’s characteristics and emotional situations, a pilot study was conducted that included 15 children with ASD and their parents and teachers. The DTT steps—memory recall and situation recognition, emotion recognition, emotion cues, facial expression recognition, and response behavior—were reliable in assessing the understanding of emotional situations when compared before and after the intervention in four categories of emotional situations, namely, happiness, sadness, anger, and fear. The results revealed that DailyConnect improves the understanding of situation-based emotions, particularly negative emotions (e.g., sadness: mean diff = −.687, sig. < .01, T = −3.866, d = 1.006; anger: mean diff = −.952, sig. < .01, T = −6.187, d = .705; fear: mean diff = −.961, sig. < .01, T = −5.522, d = .627); however, its effectiveness varied for children in different emotional situations. Furthermore, subjective feedback from participants and users (parents and teachers) provided insights into design considerations for similar mobile aids.

As communications are increasingly taking place virtually, the ability to present well online is becoming an indispensable skill. Online speakers are facing unique challenges in engaging with remote audiences. However, there has been a lack of evidence-based analytical systems for people to comprehensively evaluate online speeches and further discover possibilities for improvement. This paper introduces SpeechMirror, a visual analytics system facilitating reflection on a speech based on insights from a collection of online speeches. The system estimates the impact of different speech techniques on effectiveness and applies them to a speech to give users awareness of the performance of speech techniques. A similarity recommendation approach based on speech factors or script content supports guided exploration to expand knowledge of presentation evidence and accelerate the discovery of speech delivery possibilities. SpeechMirror provides intuitive visualizations and interactions for users to understand speech factors. Among them, SpeechTwin, a novel multimodal visual summary of speech, supports rapid understanding of critical speech factors and comparison of different speech samples, and SpeechPlayer augments the speech video by integrating visualization of the speaker's body language with interaction, for focused analysis. The system utilizes visualizations suited to the distinct nature of different speech factors for user comprehension. The proposed system and visualization techniques were evaluated with domain experts and amateurs, demonstrating usability for users with low visualization literacy and its efficacy in assisting users to develop insights for potential improvement.

Challenge is the core element of digital games. The wide spectrum of physical, cognitive, and emotional challenge experiences provided by modern digital games can be evaluated subjectively using a questionnaire, the CORGIS, which allows for a post hoc evaluation of the overall experience that occurred during game play. Measuring this experience dynamically and objectively, however, would allow for a more holistic view of the moment-to-moment experiences of players. This study, therefore, explored the potential of detecting perceived challenge from physiological signals. For this, we collected physiological responses from 32 players who engaged in three typical game scenarios. Using perceived challenge ratings from players and extracted physiological features, we applied multiple machine learning methods and metrics to detect challenge experiences. Results show that most methods achieved a detection accuracy of around 80%. We discuss in-game challenge perception, challenge-related physiological indicators and AI-supported challenge detection to inform future work on challenge evaluation.

Anomaly detection, particularly the detection of anomalous behaviors in dynamic and heterogeneous social networks, is becoming more and more crucial in real life. Traditional rule-based and feature-based methods cannot well capture the structural and temporal patterns of ever-changing user behaviors. Moreover, most of the existing works based on network embedding either rely on discretized snapshots, which have ignored accurate temporal relations among user behaviors and weakened the impact of new edges, or fail to utilize dynamic and heterogeneous information simultaneously to distinguish varying effects of new edges on existing nodes. In this paper, we propose an end-to-end continuous-time model, named Temporal Heterogeneous Graph Neural Network (THGNN), to detect anomalous behaviors (edges) in dynamic heterogeneous social networks. Specifically, the model constantly updates node embeddings by propagating the information of a new edge to its source and target nodes as well as their neighbors. In this process, heterogeneous encoders are employed to handle different types of nodes and edges. What is more, a novel dual-level distributive attention mechanism is designed to allocate the influence degree of a currently interacting node to its multiple neighbors, considering the combined effect of edge type and time interval information. That can be regarded as an extension of the classical aggregative attention mechanism in the opposite direction. Extensive experiments on four real-world datasets demonstrate that THGNN outperforms all the baselines on the task of anomalous edge detection, achieving an average AUC gain of 6% across all datasets.

Modeling the speaker variability is a key challenge for automatic speech recognition (ASR) systems. In this paper, the learning hidden unit contributions (LHUC) based adaptation techniques with compact speaker dependent (SD) parameters are used to facilitate both speaker adaptive training (SAT) and unsupervised test-time speaker adaptation for end-to-end (E2E) lattice-free MMI (LF-MMI) models. An unsupervised model-based adaptation framework is proposed to estimate the SD parameters in E2E paradigm using LF-MMI and cross entropy (CE) criterions. Various regularization methods of the standard LHUC adaptation, e.g., the Bayesian LHUC (BLHUC) adaptation, are systematically investigated to mitigate the risk of overfitting, on E2E LF-MMI CNN-TDNN and CNN-TDNN- BLSTM models. Lattice-based confidence score estimation is used for adaptation data selection to reduce the supervision label uncertainty. Experiments on the 300-hour Switchboard task suggest that, applying BLHUC in the proposed unsupervised E2E adaptation framework to byte pair encoding (BPE) based E2E LF-MMI systems consistently outperformed the baseline systems by relative word error rate (WER) reductions up to 10.5% and 14.7% on the NIST Hub5’00 and RT03 evaluation sets, and achieved the best performance in WERs of 9.0% and 9.7%, respectively. These results are comparable to the results of state-of-the-art adapted LF-MMI hybrid systems and adapted Conformer-based E2E systems.

Hand-object interaction understanding and the barely addressed novel view synthesis are highly desired in the immersive communication, whereas it is challenging due to the high deformation of hand and heavy occlusions between hand and object. In this paper, we propose a neural rendering and pose estimation system for hand-object interaction from sparse views, which can also enable 3D hand-object interaction editing. We share the inspiration from recent scene understanding work that shows a scene specific model built beforehand can significantly improve and unblock vision tasks especially when inputs are sparse, and extend it to the dynamic hand-object interaction scenario and propose to solve the problem in two stages. We first learn the shape and appearance prior knowledge of hands and objects separately with the neural representation at the offline stage. During the online stage, we design a rendering-based joint model fitting framework to understand the dynamic hand-object interaction with the pre-built hand and object models as well as interaction priors, which thereby overcomes penetration and separation issues between hand and object and also enables novel view synthesis. In order to get stable contact during the hand-object interaction process in a sequence, we propose a stable contact loss to make the contact region to be consistent. Experiments demonstrate that our method outperforms the state-of-the-art methods. Code and dataset are available in project web-page https://iscas3dv.github.io/HO-NeRF.

Learning 3D shape representation with dense correspondence for deformable objects is a fundamental problem in computer vision. Existing approaches often need additional annotations of specific semantic domain, e.g., skeleton poses for human bodies or animals, which require extra annotation effort and suffer from error accumulation, and they are limited to specific domain. In this paper, we propose a novel self-supervised approach to learn neural implicit shape representation for deformable objects, which can represent shapes with a template shape and dense correspondence in 3D. Our method does not require the priors of skeleton and skinning weight, and only requires a collection of shapes represented in signed distance fields. To handle the large deformation, we constrain the learned template shape in the same latent space with the training shapes, design a new formulation of local rigid constraint that enforces rigid transformation in local region and addresses local reflection issue, and present a new hierarchical rigid constraint to reduce the ambiguity due to the joint learning of template shape and correspondences. Extensive experiments show that our model can represent shapes with large deformations. We also show that our shape representation can support two typical applications, such as texture transfer and shape editing, with competitive performance. The code and models are available at https://iscas3dv.github.io/deformshape.

Continual text classification is an important research direction in Web mining. Existing works are limited to supervised approaches relying on abundant labeled data, but in the open and dynamic environment of Internet, involving constant semantic change of known topics and the appearance of unknown topics, text annotations are hard to access in time for each period. That calls for the technique of weakly supervised text classification (WSTC), which requires just seed words for each category and has succeed in static text classification tasks. However, there are still no studies of applying WSTC methods in a continual learning paradigm to actually accommodate the open and evolving Internet. In this paper, we tackle this problem for the first time and propose a framework, named Continual Learning for Weakly Supervised Text Classification (CL-WSTC), which can take any WSTC method as base model. It consists of two modules, classification decision with delay and seed word updating. In the former, the probability threshold for each category in each period is adaptively learned to determine the acceptance/rejection of texts. In the latter, with candidate words output by the base model, seed words are added and deleted via reinforcement learning with immediate rewards, according to an empirically certified unsupervised measure. Extensive experiments show that our approach has strong universality and can achieve a better trade-off between classification accuracy and decision timeliness compared to non-continual counterparts, with intuitively interpretable updating of seed words.

3D hand pose estimation is a challenging problem in computer vision due to the high degrees-of-freedom of hand articulated motion space and large viewpoint variation. As a consequence, similar poses observed from multiple views can be dramatically different. In order to deal with this issue, view-independent features are required to achieve state-of-the-art performance. In this paper, we investigate the impact of view-independent features on 3D hand pose estimation from a single depth image, and propose a novel recurrent neural network for 3D hand pose estimation, in which a cascaded 3D pose-guided alignment strategy is designed for view-independent feature extraction and a recurrent hand pose module is designed for modeling the dependencies among sequential aligned features for 3D hand pose estimation. In particular, our cascaded pose-guided 3D alignments are performed in 3D space in a coarse-to-fine fashion. First, hand joints are predicted and globally transformed into a canonical reference frame; Second, the palm of the hand is detected and aligned; Third, local transformations are applied to the fingers to refine the final predictions. The proposed recurrent hand pose module for aligned 3D representation can extract recurrent pose-aware features and iteratively refines the estimated hand pose. Our recurrent module could be utilized for both single-view estimation and sequence-based estimation with 3D hand pose tracking. Experiments show that our method improves the state-of-the-art by a large margin on popular benchmarks with the simple yet efficient alignment and network architectures.

Emotional information plays an important role in various multimedia applications. Movies, as a widely available form of multimedia content, can induce multiple positive emotions and stimulate people's pursuit of a better life. Different from negative emotions, positive emotions are highly correlated and difficult to distinguish in the emotional space. Since different positive emotions are often induced simultaneously by movies, traditional single-target or multi-class methods are not suitable for the classification of movie-induced positive emotions. In this paper, we proposeTransEEG, a model for multi-label positive emotion classification from a viewer's brain activities when watching emotional movies. The key features ofTransEEGinclude (1) explicitly modeling the spatial correlation and temporal dependencies of multi-channel EEG signals using the Transformer structure based model, which effectively addresses long-distance dependencies, (2) exploiting the label-label correlations to guide the discriminative EEG representation learning, for that we design an Inter-Emotion Mask for guiding the Multi-Head Attention to learn the inter-emotion correlations, and (3) constructing an attention score vector from the representation-label correlation matrix to refine emotion-relevant EEG features. To evaluate the ability of our model for multi-label positive emotion classification, we demonstrate our model on a state-of-the-art positive emotion database CPED. Extensive experimental results show that our proposed method achieves superior performance over the competitive approaches.

Benefiting from the intuitiveness and naturalness of sketch interaction, sketch-based video retrieval (SBVR) has received considerable attention in the video retrieval research area. However, most existing SBVR research still lacks the capability of accurate video retrieval with fine-grained scene content. To address this problem, in this paper we investigate a new task, which focuses on retrieving the target video by utilizing a fine-grained storyboard sketch depicting the scene layout and major foreground instances' visual characteristics (e.g., appearance, size, pose, etc.) of video; we call such a task "fine-grained scene-level SBVR". The most challenging issue in this task is how to perform scene-level cross-modal alignment between sketch and video. Our solution consists of two parts. First, we construct a scene-level sketch-video dataset called SketchVideo, in which sketch-video pairs are provided and each pair contains a clip-level storyboard sketch and several keyframe sketches (corresponding to video frames). Second, we propose a novel deep learning architecture called Sketch Query Graph Convolutional Network (SQ-GCN). In SQ-GCN, we first adaptively sample the video frames to improve video encoding efficiency, and then construct appearance and category graphs to jointly model visual and semantic alignment between sketch and video. Experiments show that our fine-grained scene-level SBVR framework with SQ-GCN architecture outperforms the state-of-the-art fine-grained retrieval methods. The SketchVideo dataset and SQ-GCN code are available in the project webpage https://iscas-mmsketch.github.io/FG-SL-SBVR/.

3D hand pose estimation from single depth is a fundamental problem in computer vision, and has wide applications. However, the existing methods still can not achieve satisfactory hand pose estimation results due to view variation and occlusion of human hand. In this paper, we propose a new virtual view selection and fusion module for 3D hand pose estimation from single depth. We propose to automatically select multiple virtual viewpoints for pose estimation and fuse the results of all and find this empirically delivers accurate and robust pose estimation. In order to select most effective virtual views for pose fusion, we evaluate the virtual views based on the confidence of virtual views using a light-weight network via network distillation. Experiments on three main benchmark datasets including NYU, ICVL and Hands2019 demonstrate that our method outperforms the state-of-the-arts on NYU and ICVL, and achieves very competitive performance on Hands2019-Task1, and our proposed virtual view selection and fusion module is both effective for 3D hand pose estimation.

Most previous EEG-based emotion recognition methods studied hand-crafted EEG features extracted from different electrodes. In this article, we study the relation among different EEG electrodes and propose a deep learning method to automatically extract the spatial features that characterize the functional relation between EEG signals at different electrodes. Our proposed deep model is calledATtention-basedLSTMwithDomainDiscriminator (ATDD-LSTM), a model based on Long Short-Term Memory (LSTM) for emotion recognition that can characterize nonlinear relations among EEG signals of different electrodes. To achieve state-of-the-art emotion recognition performance, the architecture of ATDD-LSTM has two distinguishing characteristics: (1) By applying the attention mechanism to the feature vectors produced by LSTM, ATDD-LSTM automatically selects suitable EEG channels for emotion recognition, which makes the learned model concentrate on the emotion related channels in response to a given emotion; (2) To minimize the significant feature distribution shift between different sessions and/or subjects, ATDD-LSTM uses a domain discriminator to modify the data representation space and generate domain-invariant features. We evaluate the proposed ATDD-LSTM model on three public EEG emotional databases (DEAP, SEED and CMEED) for emotion recognition. The experimental results demonstrate that our ATDD-LSTM model achieves superior performance on subject-dependent (for the same subject), subject-independent (for different subjects) and cross-session (for the same subject) evaluation.

Sketching is an intuitive and simple way to depict sciences with various object form and appearance characteristics. In the past few years, widely available touchscreen devices have increasingly made sketch-based human-AI co-creation applications popular. One key issue of sketch-oriented interaction is to prepare input sketches efficiently by non-professionals because it is usually difficult and time-consuming to draw an ideal sketch with appropriate outlines and rich details, especially for novice users with no sketching skills. Thus, sketching brings great obstacles for sketch applications in daily life. On the other hand, hand-drawn sketches are scarce and hard to collect. Given the fact that there are several large-scale sketch datasets providing sketch data resources, but they usually have a limited number of objects and categories in sketch, and do not support users to collect new sketch materials according to their personal preferences. In addition, few sketch-related applications support the reuse of existing sketch elements. Thus, knowing how to extract sketches from existing drawings and effectively re-use them in interactive scene sketch composition will provide an elegant way for sketch-based image retrieval (SBIR) applications, which are widely used in various touch screen devices. In this study, we first conduct a study on current SBIR to better understand the main requirements and challenges in sketch-oriented applications. Then we develop the SketchMaker as an interactive sketch extraction and composition system to help users generate scene sketches via reusing object sketches in existing scene sketches with minimal manual intervention. Moreover, we demonstrate how SBIR improves from composited scene sketches to verify the performance of our interactive sketch processing system. We also include a sketch-based video localization task as an alternative application of our sketch composition scheme. Our pilot study shows that our system is effective and efficient, and provides a way to promote practical applications of sketches.

Sketch-based image retrieval (SBIR) is a long-standing research topic in computer vision. Existing methods mainly focus on category-level or instance-level image retrieval. This paper investigates the fine-grained scene-level SBIR problem where a free-hand sketch depicting a scene is used to retrieve desired images. This problem is useful yet challenging mainly because of two entangled facts: 1) achieving an effective representation of the input query data and scene-level images is difficult as it requires to model the information across multiple modalities such as object layout, relative size and visual appearances, and 2) there is a great domain gap between the query sketch input and target images. We present SceneSketcher-v2, a Graph Convolutional Network (GCN) based architecture to address these challenges. SceneSketcher-v2 employs a carefully designed graph convolution network to fuse the multi-modality information in the query sketch and target images and uses a triplet training process and end-to-end training manner to alleviate the domain gap. Extensive experiments demonstrate SceneSketcher-v2 outperforms state-of-the-art scene-level SBIR models with a significant margin.

What makes speeches effective has long been a subject for debate, and until today there is broad controversy among public speaking experts about what factors make a speech effective as well as the roles of these factors in speeches. Moreover, there is a lack of quantitative analysis methods to help understand effective speaking strategies. In this paper, we propose E-ffective, a visual analytic system allowing speaking experts and novices to analyze both the role of speech factors and their contribution in effective speeches. From interviews with domain experts and investigating existing literature, we identified important factors to consider in inspirational speeches. We obtained the generated factors from multi-modal data that were then related to effectiveness data. Our system supports rapid understanding of critical factors in inspirational speeches, including the influence of emotions by means of novel visualization methods and interaction. Two novel visualizations include E-spiral (that shows the emotional shifts in speeches in a visually compact way) and E-script (that connects speech content with key speech delivery information). In our evaluation we studied the influence of our system on experts' domain knowledge about speech factors. We further studied the usability of the system by speaking novices and experts on assisting analysis of inspirational speech effectiveness.

In this paper, we propose a reconfigurable electrode, RElectrode, using a microfluidic technique that can change the geometry and material properties of the electrode to satisfy the needs for sensing a variety of different types of user input through touch/touchless gestures, pressure, temperature, and distinguish between different types of objects or liquids. Unlike the existing approaches, which depend on the specific-shaped electrode for particular sensing (e.g., coil for inductive sensing), RElectrode enables capacity, inductance, resistance/pressure, temperature, pH sensings all in a single package. We demonstrate the design and fabrication of the microfluidic structure of our RElectrode, evaluate its sensing performance through several studies, and provide some unique applications. RElectrode demonstrates technical feasibility and application values of integrating physical and biochemical properties of microfluidics into novel sensing interfaces.

In this paper, we propose a novel deep learning framework to reconstruct 3D hand poses and shapes of two interacting hands from a single color image. Previous methods designed for single hand cannot be easily applied for the two hand scenario because of the heavy inter-hand occlusion and larger solution space. In order to address the occlusion and similar appearance between hands that may confuse the network, we design a hand pose-aware attention module to extract features associated to each individual hand respectively. We then leverage the two hand context presented in interaction to propose a context-aware cascaded refinement that improves the hand pose and shape accuracy of each hand conditioned on the context between interacting hands. Extensive experiments on the main benchmark datasets demonstrate that our method predicts accurate 3D hand pose and shape from single color image, and achieves the state-of-the-art performance. Code is available in project webpage https://baowenz.github.io/Intershape/.

Relation classification (RC) is an important task in knowledge extraction from texts, while data-driven approaches, although achieving high performance, heavily rely on a large amount of annotated training data. Recently, many few-shot RC models have been proposed and yielded promising results in general domain datasets, but when adapting to a specific domain, such as medicine, the performance drops dramatically. In this paper, we propose a Knowledge-Enhanced Few-shot RC model for the Domain Adaptation task (KEFDA), which incorporates general and domain-specific knowledge graphs (KGs) to the RC model to improve its domain adaptability. With the help of concept-level KGs, the model can better understand the semantics of texts and easily summarize the global semantics of relation types from only a few instances. To be more important, as a kind of meta-information, the manner of utilizing KGs can be transferred from existing tasks to new tasks, even across domains. Specifically, we design a knowledge-enhanced prototypical network to conduct instance matching, and a relation-meta learning network for implicit relation matching. The two scoring functions are combined to infer the relation type of a new instance. Experimental results on the Domain Adaptation Challenge in the FewRel 2.0 benchmark demonstrate that our approach significantly outperforms the state-of-the-art models (by 6.63% on average).

Designing scheduling policies for multiprocessor real-time systems is challenging since the multiprocessor scheduling problem is NP-complete. The existing heuristics are customized policies that may achieve poor performance under some specific task loads. Thus, a new design pattern is needed to make the multiprocessor scheduling policies perform well under various task loads. In this paper, we investigate a new realtime scheduling policy based on reinforcement learning. For any given real-time task set, our policy can automatically derive a high performance by online learning. Specifically, we model the real-time scheduling process as a multi-agent cooperative game and propose multi-agent self-cooperative learning that overcomes the curse of dimensionality and credit assignment problems. Simulation results show that our approach can learn high-performance policies for various task/system models.

Abstract Virtual reality (VR) volleyball games can provide an immersive entertainment experience and facilitate users to get familiar with game rules. In real world, players often experience emotions from scores, strokes, passes, and the influence from their teammates or opponents, and so forth. However, seldom studies are carried out for this emotional experience in a virtual environment, as most VR volleyball games mainly concentrate on the game playing, such as body movements. In this article, we propose to enhance emotional experience by building emotional virtual characters in VR volleyball games. The virtual characters cannot only arouse their emotions but also express their facial expressions according to the game situation. Emotion patterns are learned from real‐world volleyball matches. The results demonstrate that our framework has great potential to enhance the user's emotional experience and engagement.

Chatbots systems, despite their popularity in today's HCI and CSCW research, fall short for one of the two reasons: 1) many of the systems use a rule-based dialog flow, thus they can only respond to a limited number of pre-defined inputs with pre-scripted responses; or 2) they are designed with a focus on single-user scenarios, thus it is unclear how these systems may affect other users or the community. In this paper, we develop a generalizable chatbot architecture (CASS) to provide social support for community members in an online health community. The CASS architecture is based on advanced neural network algorithms, thus it can handle new inputs from users and generate a variety of responses to them. CASS is also generalizable as it can be easily migrate to other online communities. With a follow-up field experiment, CASS is proven useful in supporting individual members who seek emotional support. Our work also contributes to fill the research gap on how a chatbot may influence the whole community's engagement.

Hand pose understanding is essential to applications such as human computer interaction and augmented reality. Recently, deep learning based methods achieve great progress in this problem. However, the lack of high-quality and large-scale dataset prevents the further improvement of hand pose related tasks such as 2D/3D hand pose from color and depth from color. In this paper, we develop a large-scale and high-quality synthetic dataset, PBRHand. The dataset contains millions of photo-realistic rendered hand images and various ground truths including pose, semantic segmentation, and depth. Based on the dataset, we firstly investigate the effect of rendering methods and used databases on the performance of three hand pose related tasks: 2D/3D hand pose from color, depth from color and 3D hand pose from depth. This study provides insights that photo-realistic rendering dataset is worthy of synthesizing and shows that our new dataset can improve the performance of the state-of-the-art on these tasks. This synthetic data also enables us to explore multi-task learning, while it is expensive to have all the ground truth available on real data. Evaluations show that our approach can achieve state-of-the-art or competitive performance on several public datasets.

Recent emerging technologies such AR/VR and HCI are drawing high demand on more comprehensive hand shape understanding, requiring not only 3D hand skeleton pose but also hand shape geometry. In this paper, we propose a deep learning framework to produce 3D hand shape from a single depth image. To address the challenge that capturing ground truth 3D hand shape in the training dataset is non-trivial, we leverage synthetic data to construct a statistical hand shape model and adopt weak supervision from widely accessible hand skeleton pose annotation. To bridge the gap due to the different hand skeleton definitions in the existing public datasets, we propose a joint regression network for hand pose adaptation. To reconstruct the hand shape, we use Chamfer loss between the predicted hand shape and the point cloud from the input depth to learn the shape reconstruction model in a weakly-supervised manner. Experiments demonstrate that our model adapts well to the real data and produces accurate hand shapes that outperform the state-of-the-art methods both qualitatively and quantitatively.

Granular media is the second-most-manipulated substance on Earth, second only to water. However, simulation of granular media is still challenging due to the complexity of granular materials and the large number of discrete solid particles. As we know, dry granular materials could form a hybrid state between a fluid and a solid, therefore we propose a two-layer model and divide the simulation domain into a dilute layer, where granules can move freely as a fluid, and a dense layer, where granules act more like a solid. Motivated by the shallow water equations, we derive a set of shallow sand equations for modeling dry granular flows by depth-integrating three-dimensional governing equations along its vertical direction. Unlike previous methods for simulating a 2D granular media, our model does not restrict the depth of the granular media to be shallow anymore. To allow efficient fluid-solid interactions, we also present a ray casting algorithm for one-way solid-fluid coupling. Finally, we introduce a particle-tracking method to improve the visual representation. Our method can be efficiently implemented based on a height field and is fully compatible with modern GPUs, therefore allows us to simulate large-scale dry granular flows in real time.

Modeling the endpoint uncertainty of moving target selection with crossing is essential to understand factors such as speed-accuracy trade-off and interaction efficiency in crossing-based user interfaces with dynamic contents. However, there have been few studies looking into this research topic in the HCI field. This paper presents a Quaternary-Gaussian model to quantitatively measure the endpoint uncertainty in crossing-based moving target selection. To validate this model, we conducted an experiment with discrete crossing tasks on five factors, i.e., initial distance, size, speed, orientation, and moving direction. Results showed that our model fit the data of μ and σ accurately with adjusted R2 of 0.883 and 0.920. We also demonstrated the validity of our model in predicting error rates in crossing-based moving target selection. We concluded with a set of implications for future designs.

On-skin haptic interfaces using soft elastomers which are thin and flexible have significantly improved in recent years. Many are focused on vibrotactile feedback that requires complicated parameter tuning. Another approach is based on mechanical forces created via piezoelectric devices and other methods for non-vibratory haptic sensations like stretching, twisting. These are often bulky with electronic components and associated drivers are complicated with limited control of timing and precision. This paper proposes HapBead, a new on-skin haptic interface that is capable of rendering vibration like tactile feedback using microfluidics. HapBead leverages a microfluidic channel to precisely and agilely oscillate a small bead via liquid flow, which then generates various motion patterns in channel that creates highly tunable haptic sensations on skin. We developed a proof-of-concept design to implement thin, flexible and easily affordable HapBead platform, and verified its haptic rendering capabilities via attaching it to users' fingertips. A study was carried out and confirmed that participants could accurately tell six different haptic patterns rendered by HapBead. HapBead enables new wearable display applications with multiple integrated functionalities such as on-skin haptic doodles, visuo-haptic displays and haptic illusions.

Recent work introduced the notion of 'emotional challenge' promising for understanding more unique and diverse player experiences (PX). Although emotional challenge has immediately attracted HCI researchers' attention, the concept has not been experimentally explored, especially in virtual reality (VR), one of the latest gaming environments. We conducted two experiments to investigate how emotional challenge affects PX when separately from or jointly with conventional challenge in VR and PC conditions. We found that relatively exclusive emotional challenge induced a wider range of different emotions in both conditions, while the adding of emotional challenge broadened emotional responses only in VR. In both experiments, VR significantly enhanced the measured PX of emotional responses, appreciation, immersion and presence. Our findings indicate that VR may be an ideal medium to present emotional challenge and also extend the understanding of emotional (and conventional) challenge in video games.

In this paper, we propose an effective and compact image-based pose representation named Poseimage Pyramid, which encodes the spatial and temporal information of human pose or hand pose as an image pyramid. Poseimage is constructed by the normalized distance between pairwise joints, and it has the advantage of its invariant to similarity transformations. With our Poseimage representation we can design the pose based action recognition or hand gesture recognition model using existing image or video classification models. In order to adapt to different actions with a variety of movement speed, we design Poseimage Pyramid to encode the multi-scale temporal information of human pose or hand pose. Experiments demonstrate that our pose representation is effective, and we achieve state-of-the-art performance on the action recognition datasets and the hand gesture recognition datasets. Our pose presentation is also complementary to video and optical flow streams in the seminal action recognition network I3D, and we achieve the state-of-the-art performance on the JHMDB, HMDB and UCF101 datasets by integrating our pose representation with I3D.

Dataless text classification has attracted increasing attentions recently. It only needs very few seed words of each category to classify documents, which is much cheaper than supervised text classification that requires massive labeling efforts. However, most of existing models pay attention to long texts, but get unsatisfactory performance on short texts, which have become increasingly popular on the Internet. In this paper, we at first propose a novel model named Seeded Biterm Topic Model (SeedBTM) extending BTM to solve the problem of dataless short text classification with seed words. It takes advantage of both word co-occurrence information in the topic model and category-word similarity from widely used word embeddings as the prior topic-in-set knowledge. Moreover, with the same approach, we also propose Seeded Twitter Biterm Topic Model (SeedTBTM), which extends Twitter-BTM and utilizes additional user information to achieve higher classification accuracy. Experimental results on five real short-text datasets show that our models outperform the state-of-the-art methods, and especially perform well when the categories are overlapping and interrelated.

Second language (L2) pronunciation training has been a worldwide task. Although computer technology makes it possible to develop a talking head to teach pronunciation like a real language teacher, little is known about how a talking head may act on L2 learners’ emotional and cognitive learning process. We investigate L2 learners’ achievement emotions, cognitive load, and pronunciation learning performance in a computer-assisted pronunciation training (CAPT) system embedded with four conditions: audio only (AU), a human face (HF), a 3D talking head with front view (3Df), and a 3D talking head with both front and profile views (3D). Results showed that, with learning time went on, participants’ perceived anxiety, boredom, and pride increased while shame and hopelessness decreased and enjoyment kept stable. With 3D, participants’ anxiety increased the most and boredom increased the least. Moreover, 3D group also perceived the highest germane load and got the highest pronunciation learning performance. Furthermore, anxiety and shame correlated with learning performance positively while boredom correlated with it negatively; enjoyment and pride correlated positively with performance on Mandarin tones. These findings significantly contribute to the efforts to design or select virtual characters for computer-aided language learning (CALL) and also provide a valuable reference to study achievement emotions in HCI systems.

Parkinson's disease (PD) is a chronic neurological disorder causing progressive disability that severely affects patients' quality of life. Although early interventions can provide significant benefits, PD diagnosis is often delayed due to both the mildness of early signs and the high requirements imposed by traditional screening and diagnosis methods. In this paper, we explore the feasibility and accuracy of detecting motor impairment in early PD via sensing and analyzing users' common touch gestural interactions on smartphones. We investigate four types of common gestures, including flick, drag, pinch, and handwriting gestures, and propose a set of features to capture PD motor signs. Through a 102-subject (35 early PD subjects and 67 age-matched controls) study, our approach achieved an AUC of 0.95 and 0.89/0.88 sensitivity/specificity in discriminating early PD subjects from healthy controls. Our work constitutes an important step towards unobtrusive, implicit, and convenient early PD detection from routine smartphone interactions.

Instant photo taking and sharing has become one of the most popular forms of social networking. However, taking high-quality photos is difficult as it requires knowledge and skill in photography that most non-expert users lack. In this paper we present SmartEye, a novel mobile system to help users take photos with good compositions in-situ. The back-end of SmartEye integrates the View Proposal Network (VPN), a deep learning based model that outputs composition suggestions in real time, and a novel, interactively updated module (P-Module) that adjusts the VPN outputs to account for personalized composition preferences. We also design a novel interface with functions at the front-end to enable real-time and informative interactions for photo taking. We conduct two user studies to investigate SmartEye qualitatively and quantitatively. Results show that SmartEye effectively models and predicts personalized composition preferences, provides instant high-quality compositions in-situ, and outperforms the non-personalized systems significantly.

Hand-drawn sketch recognition is a fundamental problem in computer vision, widely used in sketch-based image and video retrieval, editing, and reorganization. Previous methods often assume that a complete sketch is used as input; however, hand-drawn sketches in common application scenarios are often incomplete, which makes sketch recognition a challenging problem. In this paper, we propose SketchGAN, a new generative adversarial network (GAN) based approach that jointly completes and recognizes a sketch, boosting the performance of both tasks. Specifically, we use a cascade Encode-Decoder network to complete the input sketch in an iterative manner, and employ an auxiliary sketch recognition task to recognize the completed sketch. Experiments on the Sketchy database benchmark demonstrate that our joint learning approach achieves competitive sketch completion and recognition performance compared with the state-of-the-art methods. Further experiments using several sketch-based applications also validate the performance of our method.

Recent PointNet-family hand pose methods have the advantages of high pose estimation performance and small model size, and it is a key problem to get effective sample points for PointNet-family methods. In this paper, we propose a two-stage coarse to fine hand pose estimation method, which belongs to PointNet-family methods and explores a new sample point strategy. In the first stage, we use 3D coordinate and surface normal of normalized point cloud as input to regress coarse hand joints. In the second stage, we use the hand joints in the first stage as the initial sample points to refine the hand joints. Experiments on widely used datasets demonstrate that using joints as sample points is more effective and our method achieves top-rank performance.

The degree and quality of instructor-student interactions are crucial for students' engagement, retention, and learning outcomes. However, such interactions are limited in live online lectures, where instructors no longer have access to important cues such as raised hands or facial expressions at the time of teaching. As a result, instructors cannot fully understand students' learning progresses. This paper presents an explorative study investigating how presenters perceive and react to audience flow prediction when giving live-stream lectures, which has not been examined yet. The study was conducted with an experimental system that can predict audience's psychological states (e.g., anxiety, flow, boredom) through real-time facial expression analysis, and can provide aggregated views illustrating the flow experience of the whole group. Through evaluation with 8 online lectures (N_instructors=8, N_learners=21), we found such real-time flow prediction and visualization can provide value to presenters. This paper contributes a set of useful findings regarding their perception and reaction of such flow prediction, as well as lessons learned in the study, which can be inspirational for building future AI-powered system to assist people in delivering live online presentations.

With the development of virtual reality (VR) and human-computer interaction technology, how to use natural and efficient interaction methods in the virtual environment has become a hot topic of research. Gesture is one of the most important communication methods of human beings, which can effectively express users’ demands. In the past few decades, gesture-based interaction has made significant progress. This article focuses on the gesture interaction technology and discusses the definition and classification of gestures, input devices for gesture interaction, and gesture interaction recognition technology. The application of gesture interaction technology in virtual reality is studied, the existing problems in the current gesture interaction are summarized, and the future development is prospected.

Background Owing to recent advances in virtual reality (VR) technologies, effective user interaction with dynamic content in 3D scenes has become a research hotspot. Moving target selection is a basic interactive task in which the user performance research in tasks is significant to user interface design in VR. Different from the existing static target selection studies, the moving target selection in VR is affected by the change in target speed, angle and size, and lack of research on some key factors. Methods This study designs an experimental scenario in which the users play badminton under the condition of VR. By adding seven kinds of modal clues such as vision, audio, haptics, and their combinations, five kinds of moving speed and four kinds of serving angles, and the effect of these factors on the performance and subjective feelings in moving target selection in VR, is studied. Results The results show that the moving speed of the shuttlecock has a significant impact on the user performance. The angle of service has a significant impact on hitting rate, but has no significant impact on the hitting distance. The acquisition of the user performance by the moving target is mainly influenced by vision under the combined modalities; adding additional modalities can improve user performance. Although the hitting distance of the target is increased in the trimodal condition, the hitting rate decreases. Conclusion This study analyses the results of user performance and subjective perception, and then provides suggestions on the combination of modality clues in different scenarios.

Background Crossing-based target selection motion may attain less error rates and higher interactive speed in some cases. Most of the research in target selection fields are focused on the analysis of the interaction results. Additionally, as trajectories play a much more important role in crossing-based target selection compared to the other interactive techniques, an ideal model for trajectories can help computer designers make predictions about interaction results during the process of target selection rather than at the end of the whole process. Methods In this paper, a trajectory prediction model for crossing-based target selection tasks is proposed by taking the reference of a dynamic model theory. Results Simulation results demonstrate that our model performed well with regard to the prediction of trajectories, endpoints and hitting time for target-selection motion, and the average error of trajectories, endpoints and hitting time values were found to be 17.28%, 2.73mm and 11.50%, respectively.

Abstract Facial expression transfer from a single image is a challenging task and has drawn sustained attention in the fields of computer vision and computer graphics. Recently, generative adversarial nets (GANs) have provided a new approach to facial expression transfer from a single image toward target facial expressions. However, it is still difficult to obtain a sequence of smoothly changed facial expressions. We present a novel GAN‐based method for generating emotional facial expression animations given a single image and several facial landmarks for the in‐between stages. In particular, landmarks of other subjects are incorporated into a GAN model to control the generated facial expression from a latent space. With the trained model, high‐quality face images and a smoothly changed facial expression sequence can be effectively obtained, which are showed qualitatively and quantitatively in our experiments on the Multi‐PIE and CK+ data sets.

Hand detection is essential for many hand related tasks, e.g., recovering hand pose and understanding gesture. However, hand detection in uncontrolled environments is challenging due to the flexibility of wrist joint and cluttered background. We propose a convolutional neural network (CNN), which formulates in-plane rotation explicitly to solve hand detection and rotation estimation jointly. Our network architecture adopts the backbone of faster R-CNN to generate rectangular region proposals and extract local features. The rotation network takes the feature as input and estimates an in-plane rotation which manages to align the hand, if any in the proposal, to the upward direction. A derotation layer is then designed to explicitly rotate the local spatial feature map according to the rotation network and feed aligned feature map for detection. Experiments show that our method outperforms the state-of-the-art detection models on widely-used benchmarks, such as Oxford and Egohands database. Further analysis show that rotation estimation and classification can mutually benefit each other.

While instrumental ensemble playing can benefit children's music education and collaboration skill development, it requires extensive training on music and instruments, which many school children lack. To help children with limited music training experience instrumental ensemble playing, we created EnseWing, an interactive system that offers such an experience. In this paper, we report the design of the EnseWing experience and a two-month field study. Our results show that EnseWing preserves the music and ensemble skills from traditional instrumental ensemble and provides more collaboration opportunities for children.

In this paper, we propose a multi-criticality graph-based end-to-end (MCE2E) task model. This task model is abstracted from the fault diagnosing and fixing process of the industrial control systems. The task in this model consists of a collection of nodes representing mixed criticality modes. Each node is defined by a parallel directed acyclic graph, the subtasks of which are pre-allocated to multiprocessors. We propose a preliminary scheme of the multi-criticality scheduling heuristics including the intra-criticality-mode graph structure for tasks at initial criticality modes, as well as the inter-criticality-mode changes via cluster grouping with the node adaption and time partitions for recovery and isolation.

Object Tracking is an important task in Computer Vision, which has gained increasing attention from academia to industry. In this paper, we propose a real-time tracking system based on weak segmentation. Different from general tracking by detection systems, we do not classify objects into car, cat or bike, instead we just classify the image into object area and non-object area. Many tracking systems simply combine weak segmentation results with Kalman filter, which cannot track objects well when the target moves irregularly. In this paper, we use state-of-art background subtraction methods to accomplish weak segmentation and incorporates weak segmentation results with the local invariant feature to create robust object association, finally we proposal a status machine to convert temporary object association into long-term object tracking results. The experiment shows our algorithm outperforms the state-of-art tracking by weak segmentation algorithms.

Body-based motion gestures have been gaining popularity in designing interactive systems. However, theories and design guidelines on the use of body movement in design have not been fully evaluated. This article investigates human ability to perform discrete target selection tasks using stretching action through two controlled experiments. The experimental results indicate that: (1) the range of the discrete levels of depth that users can easily discriminate with arm stretching is up to 16 with full visual feedback, but is down to 4 without the feedback; (2) dwelling, a gesture with keeping a hand motionless and the cursor within a target area for a certain amount of time, may be the best gesture for confirmation command; (3) full visual feedback can improve the user performance; and (4) arm stretching action can be modeled using Fitts’ law. We also discuss the design potentials for Stretch Widgets based on these results.

Large-scale dynamic relational data visualization has attracted considerable research attention recently. We introduce dynamic data visualization into the multimedia domain, and present an interactive and scalable system, VideoMap, for exploring large-scale video content. A long video or movie has much content; the associations between the content are complicated. VideoMap uses new visual representations to extract meaningful information from video content. Map-based visualization naturally and easily summarizes and reveals important features and events in video. Multi-scale descriptions are used to describe the layout and distribution of temporal information, spatial information, and associations between video content. Firstly, semantic associations are used in which map elements correspond to video contents. Secondly, video contents are visualized hierarchically from a large scale to a fine-detailed scale. VideoMap uses a small set of sketch gestures to invoke analysis, and automatically completes charts by synthesizing visual representations from the map and binding them to the underlying data. Furthermore, VideoMap allows users to use gestures to move and resize the view, as when using a map, facilitating interactive exploration. Our experimental evaluation of VideoMap demonstrates how the system can assist in exploring video content as well as significantly reducing browsing time when trying to understand and find events of interest.

Textual documents created and distributed on the Internet are ever changing in various forms. Most of existing works are devoted to topic modeling and the evolution of individual topics, while sequential relations of topics in successive documents published by a specific user are ignored. In this paper, in order to characterize and detect personalized and abnormal behaviors of Internet users, we propose Sequential Topic Patterns (STPs) and formulate the problem of mining User-aware Rare Sequential Topic Patterns (URSTPs) in document streams on the Internet. They are rare on the whole but relatively frequent for specific users, so can be applied in many real-life scenarios, such as real-time monitoring on abnormal user behaviors. We present a group of algorithms to solve this innovative mining problem through three phases: preprocessing to extract probabilistic topics and identify sessions for different users, generating all the STP candidates with (expected) support values for each user by pattern-growth, and selecting URSTPs by making user-aware rarity analysis on derived STPs. Experiments on both real (Twitter) and synthetic datasets show that our approach can indeed discover special users and interpretable URSTPs effectively and efficiently, which significantly reflect users' characteristics.

A majority of video summarization systems use linear representations, such as rectangular storyboards and timelines at linear scales. In this paper, we propose a novel nonlinear dynamic representation called SpiralTape that summarizes a video in a smooth spiral pattern. SpiralTape provides an unusual and fresh activity suitable for stimulating environments such as science and technology museums, in which children or young individuals can have enjoyable experiences that create meaningful learning outcomes. In addition, SpiralTape provides an uninterrupted overall structure of video content and takes design principles including compactness, continuity, efficient overview, and interactivity into consideration. A working SpiralTape system was developed and deployed in pilot applications and exhibitions. Elaborate user studies with evaluation benchmarks on multiple metrics were conducted to compare SpiralTape with two representative linear video summarization methods and a state-of-the-art radial video visualization. The evaluation results demonstrate the effectiveness and natural interaction performance of SpiralTape.

Visualizing and communicating insights through maps offers an intuitive and familiar way to explore large-scale dynamic relational data. In this paper, we present VideoMap, which is a novel approach for presenting and interacting with relational video content by taking advantage of the map metaphor. VideoMap employs a metaphor to visualize video content by elements of a map with the aim of enabling exploration of video content as if reading a map. Video content is visualized in a hierarchal structure from a very large scale to a small scale of finely detailed representation. VideoMap recognizes a small set of sketch gestures for semantic zooming in and out, annotating the map, and automatically completing path navigation. To achieve this, VideoMap synthesizes map-derived visuals and binds them to the underlying data by operating the map with sketch interaction to facilitate interactive exploration. Extensive user studies were conducted to evaluate VideoMap, and the results demonstrated the effectiveness of VideoMap for facilitating the exploration and understanding of large video content.

We propose a real-time 3D model-based method that continuously recognizes dimensional emotions from facial expressions in natural communications. In our method, 3D facial models are restored from 2D images, which provide crucial clues for the enhancement of robustness to overcome large changes including out-of-plane head rotations, fast head motions and partial facial occlusions. To accurately recognize the emotion, a novel random forest-based algorithm which simultaneously integrates two regressions for 3D facial tracking and continuous emotion estimation is constructed. Moreover, via the reconstructed 3D facial model, temporal information and user-independent emotion presentations are also taken into account through our image fusion process. The experimental results show that our algorithm can achieve state-of-the-art result with higher Pearson's correlation coefficient of continuous emotion recognition in real time.

The booming of mobile devices and networks leads to an explosive growth in video resources. Efficient video research styles are appealing for facilely exploring video content appropriate to user's intention with a low cognitive load. The style of input becomes particularly relevant during the process of finding a target video clip in a large-scale database on tablets or other mobile devices. Some users have strong allegiance to input text, while others only input sketches. In this paper, we present the first systematic comparison of these two input styles and analyze the responses and feedbacks of users. An elaborated user study was conducted to test two different styles of inputting the semantics. Some users preferred to text input because it could describe their objective easily in a short time, yet some users also liked sketch because it helped illustrate the action or orientation clearly and immediately. Combining text with sketch ("sketch-text") is efficient for searching video of complex queries. The evaluation results show users' enjoying "sketch-text" and its higher performance than other input styles.

Automatic motion estimation of multiple depth cameras has remained a challenging topic in computer vision due to its reliance on the image correspondence problem. In this paper, spherical objects are employed to estimate motion parameters between multiple depth cameras. We move a sphere several times in the common view of depth cameras. We fit the spherical point clouds to get the sphere centers in each depth camera system, and then introduce a factorization based approach to estimate motions between the depth cameras. Both simulated and real experiments show the robustness and effectiveness of our method.

Although marker-based optical motion capture has been a useful method for computer animation during the past decades, automatic and robust motion tracking from multiple video sequences is still very challenging. Several critical issues in practical implementations are not adequately addressed. For example, how to track and identify the reconstructed 3D points after image matching process? How to handle the heavy occlusion problem? This paper gives a careful investigation of the above issues. In particular, we propose a novel way to track and identify proper markers, and a new method of filling missing markers by taking account of the human model constraints. Experiments are presented to show its accuracy and robustness.

In multi-touch interactive systems, it is of great significance to distinguish which hand of the user is touching the surface in real time. Left-right hand distinction is essential for recognizing the multi-finger gestures and further fully exploring the potential of bimanual interaction. However, left-right hand distinction is beyond the capability of most existing multi-touch systems. In this paper, we present a new method for left and right hand distinction based on the human anatomy, work area, finger orientation and finger position. Considering the ergonomics principles of gesture designing, the body-forearm triangle model was proposed. Furthermore, a heuristic algorithm was introduced to group multi-touch contact points and then made left-right hand distinction. A dataset of 2880 images has been set up to evaluate the proposed left-right hand distinction method. The experimental results demonstrate that our method can guarantee the high recognition accuracy and real time performance in freely bimanual multi-touch interactions.

Plain text documents created and distributed on the Internet are ever changing in various forms. Mining topics of these documents has significant applications in many domains. Most of the literature is devoted to topic modeling, while sequential patterns of topics in document streams are ignored. Moreover, traditional sequential pattern mining algorithms mainly focused on frequent patterns for deterministic data sets, and thus not suitable for document streams with topic uncertainty and rare patterns. In this paper, we formulate and handle the mining problem of rare Sequential Topic Patterns (STPs) for Internet document streams, which are rare on the whole but relatively often for specific users, so also interesting. Since this type of rare STPs reflects users’ specific behaviors, our work can be applied in many fields, such as personalized context-aware recommendation and real-time monitoring on abnormal user behaviors on the Internet. We propose a novel approach to discovering user-related rare STPs based on the temporal and probabilistic information of concerned topics. After extracting topics from documents by LDA and sorting the document stream into sessions for different users during different time periods, the proposed algorithms discover rare STPs by (1) mining STP candidates for each user through an efficient algorithm based on pattern-growth, and (2) generating user-related rare STPs by pattern rarity analysis. Experiments on both synthetic and real data sets show that our approach can discover interesting rare STPs very effectively and efficiently.

Smoothed particle hydrodynamics (SPH) is efficient, mass preserving, and flexible in handling topological changes. However, sparsely sampled thin features are difficult to simulate in SPH-based free surface flows, due to a number of robustness and stability issues. In this article, we address this problem from two perspectives: the robustness of surface forces and the numerical instability of thin features. We present a new surface tension force scheme based on a free surface energy functional, under the diffuse interface model. We develop an efficient way to calculate the air pressure force for free surface flows, without using air particles. Compared with previous surface force formulae, our formulae are more robust against particle sparsity in thin feature cases. To avoid numerical instability on thin features, we propose to adjust the internal pressure force by estimating the internal pressure at two scales and filtering the force using a geometry-aware anisotropic kernel. Our result demonstrates the effectiveness of our algorithms in handling a variety of sparsely sampled thin liquid features, including thin sheets, thin jets, and water splashes.

In China, hundreds of millions of migrant workers have moved to cities or coastal regions for more or better-paid jobs and have left their children behind at their rural homes. Separated by thousands of kilometers, these "left-behind" children and their migrant parents use mobile phones as their primary - and often only - method of maintaining family connections. To better understand the use of technology in this long-distance communication, we conducted a multi-phased study using interviews and surveys in three different Chinese rural areas. In this paper, we report our findings on how these children communicate with their migrant parents and what information they exchange. We also discuss design implications derived from these findings that may improve communication between left-behind children and their parents.

Semi-Markov conditional random fields (semi-CRFs) are usually trained with maximum a posteriori (MAP) criterion which adopts the 0/1 cost for measuring the loss of misclassification. In this paper, based on our previous work on handwritten Chinese/Japanese text recognition (HCTR) using semi-CRFs, we propose an alternative parameter learning method by minimizing the risk, in which the misclassification costs are not equal, but different depending on the hypothesis and the ground-truth. The proposed method is lattice-based, i.e., the hypothesis space is the entire lattice on which the semi-CRF is defined. Experimental results on two online handwriting databases: CASIA-OLHWDB and TUAT Kondate demonstrate that minimum-risk training can yield superior string recognition rates compared to MAP training.

A low cost on-line admission controller is required by hard real-time system working in dynamic circumstances. In this paper, we propose a new utilization based constant-time admission control algorithm, called AC for aperiodic tasks under EDF scheduling. We prove that given the same processor state, AC is safe and has stronger admission capability than the best existing utilization-based admission control algorithm with constant-time complexity. Simulation results show that AC also has good performance in success ratio and efficiency.

Sparse bundle adjustment is widely used in many computer vision applications. In this paper, we propose a method for performing bundle adjustments using the L1norm. After linearizing the mapping function in bundle adjustment on its first order, the kernel step is to compute the L1norm equations. Considering the sparsity of the Jacobian matrix in linearizing, we find two practical methods to solve the L1norm equations. The first one is an interior-point method, which transfer the original problem to a problem of solving a sequence of L2norm equations, and the second one is a decomposition method which uses the differentiability of linear programs and represents the optimal updating of parameters of 3D points by the updating variables of camera parameters. The experiments show that the method performs better for both synthetically generated and real data sets in the presence of outliers or Laplacian noise compared with the L2norm bundle adjustment, and the method is efficient among the state of the art L1minimization methods.

In this paper, we introduce the design and implementation of TempoString, an easy-to-use tool which assists children with music creation. It provides such a fun and novel platform by allowing children to "draw" music on a canvas and then edit it using a rope. The main contribution of our work is the novel access which allows children to "paint" music on a canvas and then edit using a rope.

Gesture inputs on multi-touch tabletops usually involve multiple fingers (more than two) and casual touchdowns or liftoffs of fingers. This flexibility of touch gestures allows more natural user interaction, but also poses new challenges for accurate recognition of multi-touch gestures. To address these challenges, we propose a new approach to recognize flexible multi-touch stroke gestures on tabletops. Based on a user study on multi-touch unistroke gestures, we develop a gesture recognition method by extracting key strokes embedded in flexible multi-touch input. Our evaluation study result shows that this method can greatly improve the recognition accuracy of flexible multi-touch unistroke gestures on tabletops.

Abstract Enforcing fluid incompressibility is one of the time‐consuming aspects in SPH. In this paper, we present a local Poisson SPH (LPSPH) method to solve incompressibility for particle based fluid simulation. Considering the pressure Poisson equation, we first convert it into an integral form, and then apply a discretization to convert the continuous integral equation to a discretized summation over all the particles in the local pressure integration domain determined by the local geometry. To control the approximation error, we further integrate our local pressure solver into the predictive‐corrective framework to avoid the computational cost of solving a pressure Poisson equation globally. Our method can effectively eliminate the large density deviations mainly caused by the solid boundary treatment and free surface topological change, and show advantage of a higher convergence rate over the predictive‐corrective incompressible SPH (PCISPH).

Authoring context-aware, interactive video representation is usually a complex process. A user-friendly multimedia authoring environment is thus solicited to explore and express users' design ideas efficiently and naturally. In this paper we present a sketch-based two-layer representation, called scene structure graph (SSG), to facilitate the video authoring process. One layer in SSG uses sketches as a concise form with which the visualization of scene information is easily understood and the other layer uses a graph to represent and edit the narrative structure in the authoring process. With SSG, the authoring process works in two stages. In the first stage, various sketch forms such as symbols and hand-drawing illustrations are used as basic primitives to annotate the video clips and the hyperlinks encoding spatio-temporal relations are established in SSG. In the second stage, sketches in SSGs are modified and new SSG is composed for any particular authoring purpose. Three user studies are elaborated, showing that the SSG is user-friendly and can achieve a good balance between expressiveness of users' intent and ease of use for authoring of interactive video.

Simulating solid-fluid coupling with the classical meshless methods is an difficult issue due to the lack of the Kronecker delta property of the shape functions when enforcing the essential boundary conditions. In this work, we present a novel staggered meshless method to overcome this problem. We create a set of staggered particles from the original particles in each time step by mapping the mass and momentum onto these staggered particles, aiming to stagger the velocity field from the pressure field. Based on this arrangement, an new approximate projection method is proposed to enforce divergence-free on the fluid velocity with compatible boundary conditions. In the simulations, the method handles the fluid and solid in a unified meshless manner and generalizes the formulations for computing the viscous and pressure forces. To enhance the robustness of the algorithm, we further propose a new framework to handle the degeneration case in the solid-fluid coupling, which guarantees stability of the simulation. The proposed method offers the benefit that various slip boundary conditions can be easily implemented. Besides, explicit collision handling for the fluid and solid is avoided. The method is easy to implement and can be extended from the standard SPH algorithm in a straightforward manner. The paper also illustrates both one-way and two-way couplings of the fluids and rigid bodies using several test cases in two and three dimensions.

This paper presents a tangible programming tool 'T-Maze' for children aged 5 to 9. Children could use T-Maze to create their own maze maps and complete some maze escaping tasks by the tangible programming blocks and sensors. T-Maze uses a camera to, in real-time, catch the programming sequence of the wooden blocks' arrangement, which will be used to analyze the semantic correctness and enable the children to receive feedbacks immediately. And children could join in the game by controlling the sensors during program's running. A user study shows that T-Maze is an interesting programming approach for children and easy to learn and use.

The classical Mean Shift can't change the scale of tracking window in real time while tracking target is changing in size. This paper adopts graph cuts theory to the problem of scale adaptation for Mean Shift tracking. According to the result of Mean Shift iteration in every frame, implementing graph cuts using skin color Gaussian mixture model(GMM) in a small area around it, and updating tracking window size through the largest skin lump among the result of graph cuts. Experimental results clearly demonstrate that the method can reflect the real scale change of tracking target, avoid the interference of other objects in background, and has good usability and robustness. Besides it enriches manipulation method of Human Computer Interaction by controlling entertainment games.

With the fast economic growth and urbanization of many developing countries come concerns that their children now have fewer opportunities to express creativity and develop collaboration skills, or to experience their local cultural heritage. We propose to address these concerns by creating technologies inspired by traditional arts, and allowing children to create and collaborate through playing with them. ShadowStory is our first attempt in this direction, a digital storytelling system inspired by traditional Chinese shadow puppetry. We present the design and implementation of ShadowStory and a 7-day field trial in a primary school. Findings illustrated that ShadowStory promoted creativity, collaboration, and intimacy with traditional culture among children, as well as interleaved children's digital and physical playing experience.

In this paper, we describe CoolMag, a tangible interaction tool to enable children to create different instruments collaboratively in music education. With CoolMag, children could learn the basic playing methods of different instruments. It also has the potential to inspire children's creativity, because children could adopt objects in daily life (broom, cup, pen etc.) as the carrier of their novel instruments whose appearance may differ from the traditional one.

AID object is a segment with several known-distance markers, and calibration methods with ID objects are more flexible than those with 2D/3D objects. Under the pinhole camera model, it is proved that the calibration with free-moving ID objects is not possible. For a central catadioptric camera setup, can the camera be calibrated by a ID object under general motions? In this paper, we prove that a central catadioptric camera can indeed be calibrated, and propose a catadioptric camera calibration method using ID objects undertaking general motions. The proposed method consists of two steps. Firstly, the principal point is calculated with geometric invariants under catadioptric camera model; Secondly, we use images of ID object to calibrate the focal lengths, skew factor and mirror parameter. The method needs neither prior knowledge of catadioptric parameters nor conic fitting, and it is linear, which makes it easy to implement. Experiments demonstrate its usefulness and stability.

In this paper, we introduce an approach to visualize the inference process in a rule engine -- Drools, which employs Rete as its pattern matching algorithm. As a software visualization work, our approach is focused on both static structure of the Rete network and dynamic behavior of the inference process. Since logic programming is distinct from other traditional programming paradigms, our approach is also different from traditional program/algorithm visualization methods. In this paper, we first introduce the target we choose to visualize, and then provide a description of the problem and our visualization approach. Finally, with an implementation and an interesting case -- sudoku solving, we show that the visualization work is helpful to understanding not only the Rete algorithm, but also the rules used in the inference. Besides, our work supports debugging, tracing and analyzing the rule engine, which is useful in finding errors and optimization.

Recently, pen tilting has been explored in pen-based user interfaces and has shown potential to improve user interaction in various tasks (e.g., menu selection, modeless object manipulation). However, some basic questions concerning pen tilting behaviors, such as the ideal range, azimuth size, and direction of pen tilting, have not been thoroughly investigated. In this paper, we report our empirical studies on user performances in basic pen tilting tasks. First, we conducted a baseline study, which helps us to determine tilting directions, tilting ranges, and the thresholds that separate incidental pen tilting actions from intentional actions used for interaction. Based on the results from the baseline study, we designed an experiment to investigate user performances in goal tilting in different tilting ranges, azimuth sizes, and directions. Drawing on the results of our data analyses on task completion time, error rate, and pen tip movements, we discussed values of tilting parameters like titling range, minimal azimuth size, and tilting direction.

Along with the rapid development of information technology, individual users of computers are faced with many problems in dealing with massive files, such as enormous amount, serious redundancy and complex version evolution relation. These lead that the file system becomes more and more rough-and-tumble, to a certain extent putting users to trouble in daily work. This paper systematically analyzed the problems in file system management, put forward user cognitive model when faced with massive files, and then introduced the FDTVAS system which is a visual interface based on hyperbolic tree view for exploring the massive files. This system used specific algorithm to do the intelligent analysis of file relationship, took hyperbolic tree as the main visual form, and provided a series of interaction tasks according with users' cognitive model. The built system can help users get a clear insight of the file system and do the analysis and management efficiently.

In this paper, we present a new sketch-based system - KnitSketch, to improve the efficiency of process planning for knitting garments at an early design stage. The KnitSketch system utilizes sketching interface with the pen-paper metaphor and users only need to draw outlines of different parts of the garment. Based on sketching understanding, the system automatically makes reasonable geometric inferences about the process-planning data of the garment. The system is designed for nonprofessional users and can design diverse garment styles by freehand drawings. The contributions of this work include contextual extraction of reusable data from sketches, a MDG structure for sketch beautification, and an integrated system with natural expression and effective communication that reduces the cognitive load of human beings. User experience shows that the proposed system helps designers focus on the task instead of the designing tools, and thus improves the efficiency and productivity of human beings.

Concept maps are an important tool to organize, represent, and share knowledge. Building a concept map involves creating text-based concepts and specifying their relationships with line-based links. Current concept map tools usually impose specific task structures for text and link construction, and may increase cognitive burden to generate and interact with concept maps. While pen-based devices (e.g., tablet PCs) offer users more freedom in drawing concept maps with a pen or stylus more naturally, the support for hand-drawn concept map creation and manipulation is still limited, largely due to the lack of methods to recognize the components and structures of hand-drawn concept maps. This article proposes a method to understand hand-drawn concept maps. Our algorithm can extract node blocks, or concept blocks, and link blocks of a hand-drawn concept map by combining dynamic programming and graph partitioning, recognize the text content of each concept node, and build a concept-map structure by relating concepts and links. We also design an algorithm for concept map retrieval based on hand-drawn queries. With our algorithms, we introduce structure-based intelligent manipulation techniques and ink-based retrieval techniques to support the management and modification of hand-drawn concept maps. Results from our evaluation study show high structure recognition accuracy in real time of our method, and good usability of intelligent manipulation and retrieval techniques.

In many developing countries such as India and China, low educational levels often hinder economic empowerment. In this paper, we argue that mobile learning games can play an important role in the Chinese literacy acquisition process. We report on the unique challenges in the learning Chinese language, especially its logographic writing system. Based on an analysis of 25 traditional Chinese games currently played by children in China, we present the design and implementation of two culturally inspired mobile group learning games, Multimedia Word and Drumming Strokes. These two mobile games are designed to match Chinese children's understanding of everyday games. An informal evaluation reveals that these two games have the potential to enhance the intuitiveness and engagement of traditional games, and children may improve their knowledge of Chinese characters through group learning activities such as controversy, judgments and self-correction during the game play.

Research on complex shape recognition showed that the shape context algorithm is sensitive to relative position variation of articulation. Aimed at this problem, a shape recognition method is proposed based on local shape filling rate of various object silhouettes. We take each landmark point as a circle center and use as its radius. Then, under a particular radius, the ratio between the covered silhouette pixels and the total pixels is defined as local shape filling rate. Thus, different radius may form different local shape filling rates. All landmark points with different radius will constitute a characteristic matrix which can effectively reflects the entire statistical property of the object shape. Experiments on a variety of shape databases show that the novel method is insensitive to articulation and less influenced by the number of landmark points, so our algorithm has strong power in describing object details.

Thermal management is critical for clusters because of the increasing power consumption of modern processors, compact server architectures and growing server density in data centers. Thermal balancing mitigates hot spots in a cluster through dynamic load distribution among servers. This paper presents two Control-theoretical Thermal Balancing (CTB) algorithms that dynamically balance the temperatures of different servers based on online measurements. CTB features controllers rigorously designed based on optimal control theory and a difference equation model that approximates the thermal dynamics of clusters. Control analysis and simulation results demonstrate that CTB achieves robust thermal balancing under a wide range of uncertainties: (1) when different tasks incur different power consumptions on the CPUs, (2) when servers experience different ambient temperatures, and (3) when servers experience thermal faults.

Computer-based geometry systems have been widely used for teaching and learning, but largely based on mouse-and-keyboard interaction, these systems usually require users to draw figures by following strict task structures defined by menus, buttons, and mouse and keyboard actions. Pen-based designs offer a more natural way to develop geometry theorem proofs with hand-drawn figures and scripts. This paper describes a pen-based geometry theorem proving system that can effectively recognize hand-drawn figures and hand-written proof scripts, and accurately establish the correspondence between geometric components and proof steps. Our system provides dynamic and intelligent visual assistance to help users understand the process of proving and allows users to manipulate geometric components and proof scripts based on structures rather than strokes. The results from evaluation study show that our system is well perceived and users have high satisfaction with the accuracy of sketch recognition, the effectiveness of visual hints, and the efficiency of structure-based manipulation.

Thermal control is crucial to real-time systems as excessive processor temperature can cause system failure or unacceptable performance degradation due to hardware throttling. Real-time systems face significant challenges in thermal management as they must avoid processor overheating while still delivering desired real-time performance. Furthermore, many real-time systems must handle a broad range of uncertainties in system and environmental conditions. To address these challenges, this paper presents Thermal Control under Utilization Bound (TCUB), a novel thermal control algorithm specifically designed for real-time systems. TCUB employs a nested feedback loop that dynamically controls both processor temperature and CPU utilization through task rate adaptation. Rigorously modeled and designed based on control theory, TCUB can maintain both desired processor temperature and CPU utilization, thereby avoiding processor overheating and maintaining desired soft real-time performance. A salient feature of TCUB lies on its capability to handle a broad range of uncertainties in terms of processor power consumption, task execution times, ambient temperature, and unexpected thermal faults. The robustness of TCUB makes it particularly suitable for real-time embedded systems that must operate in highly unpredictable environments. The advantages of TCUB are demonstrated through extensive simulations under a broad range of system and environmental uncertainties.

This article presents UCFM, a user-centered developing framework for interactive temporal data visualization. UCFM is comprised mainly by software architecture and software development methods. The software architecture describes modules in interactive temporal data visualization system and their relationships. And based on the software architecture and development practice, the software development methods summarize these specific steps to design and develop interactive temporal data visualization system. To demonstrate UCFM's validity, the development process of an interactive temporal data visualization application is illustrated.

Mining frequent itemsets over online data streams, where the new data arrive and the old data will be removed with high speed, is a challenge for the computational complexity. Existing approximate mining algorithms suffer from explosive computational complexity when decreasing the error parameter, isin, which is used to control the mining accuracy. We propose a new approximate mining algorithm using an approximate frequent itemset tree (abbreviated as AFI-tree), called AFI algorithm, to mine approximate frequent itemsets over online data streams. The AFI-tree based on prefix tree maintains only frequent itemsets, so the number of nodes in the tree is very small. All the infrequent child nodes of any frequent node are pruned and the maximal support of the pruned nodes is estimated to detect new frequent itemsets. In order to guarantee the mining accuracy, when the estimated maximal support of the pruned nodes is a bit more than the minimum support, their supports will be re-computed and the frequent nodes among them will be inserted into the AFI-tree. Experimental results show that the AFI algorithm consumes much less memory space than existing algorithms, and runs much faster than existing algorithms in most occasions.

Concept maps are an important tool to knowledge organization, representation, and sharing. Most current concept map tools do not provide full support for hand-drawn concept map creation and manipulation, largely due to the lack of methods to recognize hand-drawn concept maps. This paper proposes a structure recognition method. Our algorithm can extract node blocks and link blocks of a hand-drawn concept map by combining dynamic programming and graph partitioning and then build a concept-map structure by relating extracted nodes and links. We also introduce structure-based intelligent manipulation technique of hand-drawn concept maps. Evaluation shows that our method has high structure recognition accuracy in real time, and the intelligent manipulation technique is efficient and effective.

We present a new technique called 'Tilt Menu' for better extending selection capabilities of pen-based interfaces. The Tilt Menu is implemented by using 3D orientation information of pen devices while performing selection tasks. The Tilt Menu has the potential to aid traditional one-handed techniques as it simultaneously generates the secondary input (e.g., a command or parameter selection) while drawing/interacting with a pen tip without having to use the second hand or another device. We conduct two experiments to explore the performance of the Tilt Menu. In the first experiment, we analyze the effect of parameters of the Tilt Menu, such as the menu size and orientation of the item, on its usability. Results of the first experiment suggest some design guidelines for the Tilt Menu. In the second experiment, the Tilt Menu is compared to two types of techniques while performing connect-the-dot tasks using freeform drawing mechanism. Results of the second experiment show that the Tilt Menu perform better in comparison to the Tool Palette, and is as good as the Toolglass.

Complex distributed real-time applications require complicated processing and sharing of an extensive amount of data under critical timing constraints. In this paper, we present a comprehensive overview of the Data Distribution Service standard (DDS) and describe its QoS (Quality of Service) features for developing real-time applications. Real-time ECA (RECA) rules are introduced to efficiently describe QoS policy in an active real-time database (ARTDB) named Agilor. And then we propose a novel QoS-Enable Real-Time Publish-Subscribe (QERTPS) service compatible to DDS for distributed real-time data acquisition. QERTPS could support several different QoS levels for various applications at the same time. Furthermore, QERTPS is implemented by object models and RECA rules in Agilor. To illustrate the benefits of QERTPS for real-time data acquisition, an example application is presented. Experimental evaluation shows that the proposed service provides a stable and timely service for providing different QoS levels.

Chinese text entry is challenging on mobile devices which rely on keypad input. Entering one character may require many key presses. This paper proposes a multimodal text entry technique for Chinese. In this method, Chinese user can enter Chinese text by simultaneously using the simplified phonemic input method named Jianpin with keypad and speech utterance. The key of the technique is a multimodal fusion algorithm, which synchronizes speech and keypad input and fuses redundant information from two modalities to get the best candidate. A preliminary evaluation shows that users appreciate this technique and it could reduce key presses and enhance the input efficiency.

In order to improve stimulus-response compatibility of touchpad in pen-based user interface, we present the tilt cursor, i.e. a cursor dynamically reshapes itself to providing the 3D orientation cue of pen. We also present two experiments that evaluate the tilt cursor's performance in circular menu selection and specific marking menu selection tasks. Results show that in a specific marking menu selection task, the tilt cursor significantly outperforms the shape-fixed arrow cursor and the live cursor [4]. In addition, results show that by using the tilt cursor, the response latencies for adjusting drawing directions are smaller than that by using the other two kinds of cursors.

In recognition-based user interface, users' satisfaction is determined not only by recognition accuracy but also by effort to correct recognition errors. In this paper, we introduce a crossmodal error correction technique, which allows users to correct errors of Chinese handwriting recognition by speech. The focus of the paper is a multimodal fusion algorithm supporting the crossmodal error correction. By fusing handwriting and speech recognition, the algorithm can correct errors in both character extraction and recognition of handwriting. The experimental result indicates that the algorithm is effective and efficient. Moreover, the evaluation also shows the correction technique can help users to correct errors in handwriting recognition more efficiently than the other two error correction techniques.

In this paper a novel approach is described for tree visualization using nested circles. The brother nodes at the same level are represented by externally tangent circles; the tree nodes at different levels are displayed by using 2D nested circles or 3D nested cylinders. A new layout algorithm for tree structure is described. It provides a good overview for large data sets. It is easy to see all the branches and leaves of the tree. The new method has been applied to the visualization of file systems.

Co-CreativePen toolkit is a pen-based 3D toolkit for children cooperatively designing virtual environment. This toolkit is used to construct different applications involved with distributed pen-based 3D interaction. In this toolkit, sketch method is encapsulated as kinds of interaction techniques. Children can use pen to construct 3D and IBR objects, to navigate in the virtual world, to select and manipulate virtual objects, and to communicate with other children. Children can use pen to select other children in the virtual world, and use pen to write message to children selected. The distributed architecture of Co-CreativePen toolkit is based on the CORBA. A common scene graph is managed in the server with several copies of this graph are managed in every client. Every changes of the scene graph in client will cause the change in the server and other client

Recent years have seen rapid growth of online services that rely on large-scale server clusters to handle high volume of requests. Such clusters must adaptively control the CPU utilizations of many processors in order to maintain desired soft real-time performance and prevent system overload in face of unpredictable workloads. This paper presents DUC-LB, a novel distributed utilization control algorithm for cluster-based soft real-time applications. Compared to earlier works on utilization control, a distinguishing feature of DUC-LB is its capability to handle system dynamics caused by load balancing, which is a common and essential component of most clusters today. Simulation results and control-theoretic analysis demonstrate that DUCLB can provide robust utilization control and effective load balancing in large-scale clusters.

Cooperative design is one of the most important subjects of the researches on CSCW. Distributed cognition is a useful theoretical framework applied to analyzing, designing, implementing, and evaluating cooperative design systems. In this paper, we construct a detailed distributed cognition framework, explain these terms in it, and implement a prototype of the cooperative design system, smart cooperative design system (SCDS), based on the framework. In SCDS, the computer can infer not only local participant intent but also team intent by capturing information distributed external environment and individual historical interactions. Cooperative participants can learn the intent each other through information representation distributed over individuals and artifacts. In some extent, this avoids some collaborative conflict.

Innovation on 3D shape design relies on advanced product design theory and effective design tools. The requirements for design tools mainly include collaborative performance and natural performance of interaction. In this paper, a 3D collaborative editing environment based on interactive transaction log is put forward and moreover the system architecture is presented. The collaborative mechanism is discussed and the collision detection of manipulation and application strategy during the course of multi-user concurrency editing is studied, the interactive gesture appropriate to 3D model and application methods are studied too. Finally a prototype system is built, which obtains satisfied result in practice.

Summary form only given. Many researches have been devoted to designing appropriate concurrency control algorithms for real-time database systems, which not only satisfy consistency requirements but also meet transaction timing constraints as much as possible. Optimistic concurrency control protocols have the nice properties of being nonblocking and deadlock-free, but they have the problems of late conflict detection and transaction restarts. Although the number of transaction restarts is reduced by dynamic adjustment of serialization order (DASO) in real-time database systems, it still has some unnecessary transaction restarts. We first propose a new method called dynamic adjustment of execution order (DAEO) and a new optimistic concurrency control algorithm based on DAEO, which can reduce the number of unnecessary restarts near to zero and outperforms the previous algorithms, and then discuss the experiments and the results.

Ubiquitous computing provides the new research challenges for the next generation interactions. Natural User Interface is one of the important aspects. Pen and paper are the natural and simple tools to use in our daily life, which can fit for the imprecise and flexible design mode in conceptual design. Gestures are the useful and natural mode in the pen-based interaction which can be introduced and used through the similar mode to the existing resources in interaction. Inherent problems exist with collaborative conceptual design because the simultaneous group interaction required for users to smoothly and effectively work together in the same virtual space. In this paper, we provide solutions to these problems by devising a fast and direct gesture-based user interface, a set of visual effects that better enable a user's awareness of the operations done by other participants, and a set of tools for enhancing visual communication between participants. A gesture description language (GDL) is given to provide the useful gestures to the applications conveniently.

Conceptual design has played a very important role in the design process. Knowledge of all the design requirements and constraints during this early phase of a product's life cycle is usually approximate, imprecise, and unknown. Conceptual design is now recognized as a highly important process when considering both the concurrent engineering aspects and the commitment of resources necessary to design and develop a product. With the development of the Internet, collaborative design is imperative and effective. The technology of CSCW is adopted widely and necessarily in the domain of conceptual design. It can fit the nature of the design. It shortens the distance of designers in different places and collects and shares all the information in time. In conceptual design, collaborative design mainly includes function, principle, layout and initial structure, etc. We developed a system of conceptual design based on the current CAD system, which realized the seamless link with later detail design. For individual design, multi-modal technology, which integrates many interactions such as gesture and speech, supports an advanced interface to assist humans in developing applications.