Embodied reference understanding is crucial for intelligent agents to predict referents based on human intention through gesture signals and language descriptions. This paper introduces the Attention-Dynamic DINO, a novel framework designed to mitigate misinterpretations of pointing gestures across various interaction contexts. Our approach integrates visual and textual features to simultaneously predict the target object’s bounding box and the attention source in pointing gestures. Leveraging the distance-aware nature of nonverbal communication in visual perspective taking, we extend the virtual touch line mechanism and propose an attention-dynamic touch line to represent referring gesture based on interactive distances. The combination of this distance-aware approach and independent prediction of the attention source, enhances the alignment between objects and the gesture represented line. Extensive experiments on the YouRefit dataset demonstrate the efficacy of our gesture information understanding method in significantly improving task performance. Our model achieves 76.3% accuracy at the 0.25 IoU threshold and, notably, surpasses human performance at the 0.75 IoU threshold, marking a first in this domain. Comparative experiments with distance-unaware understanding methods from previous research further validate the superiority of the Attention-Dynamic Touch Line across diverse contexts.

Deep learning (DL)-based human activity recognition (HAR) methods have shown promise in the applications of health Internet of Things (IoT) and wireless body sensor networks (BSNs). However, adapting these methods to new users in real-world scenarios is challenging due to the cross-subject issue. To solve this issue, we propose ActiveSelfHAR, a framework that combines active learning’s benefit of sparsely acquiring informative samples with actual labels and self-training’s benefit of effectively utilizing unlabeled data to adapt the HAR model to the target domain, i.e., the new users. ActiveSelfHAR consists of several key steps. First, we utilize the model from the source domain to select and label the domain invariant samples, forming a self-training set. Second, we leverage the distribution information of the self-training set to identify and annotate samples located around the class boundaries, forming a core set. Third, we augment the core set by considering the spatiotemporal relationships among the samples in the nonself-training set. Finally, we combine the self-training set and augmented core set to construct a diverse training set in the target domain and fine-tune the HAR model. Through leave-one-subject-out validation on three IMU-based data sets and one EMG-based data set, our method achieves mean HAR accuracies of 95.20%, 82.06%, 89.52%, and 92.82%, respectively. Our method demonstrates similar HAR accuracies to the upper bound, i.e., fine-tuning framework with approximately 1% labeled data of the target data set, while significantly improving data efficiency and time cost. Our work highlights the potential of implementing user-independent HAR methods into health IoT and BSN.

Brain decoding that classifies cognitive states using the functional fluctuations of the brain can provide insightful information for understanding the brain mechanisms of cognitive functions. Among the common procedures of decoding the brain cognitive states with functional magnetic resonance imaging (fMRI), extracting the time series of each brain region after brain parcellation traditionally averages across the voxels within a brain region. This neglects the spatial information among the voxels and the requirement of extracting information for the downstream tasks. In this study, we propose to use a fully connected neural network that is jointly trained with the brain decoder to perform an adaptively weighted average across the voxels within each brain region. We perform extensive evaluations by cognitive state decoding, manifold learning, and interpretability analysis on the Human Connectome Project (HCP) dataset. The performance comparison of the cognitive state decoding presents an accuracy increase of up to 5% and stable accuracy improvement under different time window sizes, resampling sizes, and training data sizes. The results of manifold learning show that our method presents a considerable separability among cognitive states and basically excludes subject-specific information. The interpretability analysis shows that our method can identify reasonable brain regions corresponding to each cognitive state. Our study would aid the improvement of the basic pipeline of fMRI processing.

The prevalence of smart devices encourages increasing requirements of wearable human–computer interactions. To improve user acceptance, such interactions require easy-to-manipulate and unobtrusive characteristics. In this article, we, for the first time, propose to recognize silent commands through a lightweight and around-ear biosensing system Mordo that can be easily integrated with earphones, manipulate smart devices, and minimize social awkwardness. In particular, we first determine the empirical principles of constructing commands and experimentally screen the commands based on the around-ear configuration. Second, we select the optimal around-ear sensor configuration according to the single-channel signal-to-noise ratios (SNRs) and classification accuracies. Third, we propose a multistream CNN-LSTM network to learn the spatiotemporal mapping between the around-ear signals and commands. Finally, extensive experiments have been conducted to evaluate the feasibility and stability. The results indicate an averaged accuracy of 89.66% that outperforms other algorithms of similar tasks. The stability tests show that our system presents sufficient stability under command deformations and head motions. We demonstrate the necessity of collecting such scale of data by gradually reducing training data size. We also validate the generalization ability of our method toward other sensing parameters by reducing the spatial and temporal resolutions. The proof-of-concept design will aim the further development of the commercial products for silent command recognition.

Traceability link recovery (TLR) is an important software engineering task for developing trustworthy and reliable software systems. Recently proposed deep learning (DL) models have shown their effectiveness compared to traditional information retrieval-based methods. DL often heavily relies on sufficient labeled data to train the model. However, manually labeling traceability links is time-consuming, labor-intensive, and requires specific knowledge from domain experts. As a result, typically only a small portion of labeled data is accompanied by a large amount of unlabeled data in real-world projects. Our hypothesis is that artifacts are semantically similar if they have the same linked artifact(s). This paper presents TRACEFUN, a new approach to enhance traceability link recovery with unlabeled data. TRACEFUN first measures the similarities between unlabeled and labeled artifacts using two similarity prediction methods (i.e., vector space model and contrastive learning). Then, based on the similarities, newly labeled links are generated between the unlabeled artifacts and the linked objects of the labeled artifacts. Generated links are further used for TLR model training. We have evaluated TRACEFUN on three GitHub projects with two state-of-the-art DL models (i.e., Trace BERT and TraceNN). The results show that TRACEFUN is effective in terms of a maximum improvement of F1-score up to 21% and 1,088%, respectively for Trace BERT and TraceNN.

The cybertwin-driven 6G that can obtain static and dynamic data stream of users provide an exciting potential for a novel muscular human cybertwin beyond traditonally used artificial neural networks (ANNs) and musculoskeletal models (MSMs). In this article, we propose the conceptual design of the muscular human cybertwin and construct a baseline model with an improved generalization ability over ANN and an easier adaptation to new data distributions over MSMs. In particular, we for the first time propose to combine ANN and MSM, which benefits from the combination of learning-based approaches and analytical approaches. We then experimentally compare different manners of the combination and demonstrate the better combining manner on our testing case. Finally, we evaluate our method on an open-sourced dataset and on data from wearable sensors from the aspects of joint moment prediction accuracy, data efficiency, generalization ability, and time efficiency of personalization. Our proposed method achieves accuracy similar with ANN and over 30$\%$better than MSM with sufficient training data. Compared with ANN, the improved data efficiency is presented by the better accuracies with a small amount of training data, and the generalization ability to unseen walking conditions and new subjects are demonstrated by the over 70$\%$accuracy improvements. Moreover, when fine-tuning the model, our algorithm is demonstrated by the time 75$\%$shorter than calibrated MSM and the accuracy improvements.

The surface electromyography (sEMG) signal-based human-machine interface (HMI) has been widely used for various scenarios of physical human-robot interaction. However, current HMIs based on bipolar myoelectric sensors are hindered by the limitations of global sEMG features, which are prone to variability and delay. In this letter, we define a HMI that takes advantage of the underlying neural information of spinal module activations from bipolar sEMG signals, inspired by recent findings of neural codes. Firstly, the spinal module activations are identified by the spiking trains of the muscle synergies extracted from bipolar sEMG signals. Secondly, we extract the information encoded in both firing rates and spike timings of the spinal module activation in a population coding manner, which follows the information encoding principle of neurons. Thirdly, we map the series of spinal module activations into gait phases, locomotion modes, joint moment and human identity in order to experimentally reveal the physiological information contained in the spinal module activations. The contained information and the benefit of our design are demonstrated and experimentally explained by the presented results and comparisons with the traditionally used global sEMG features. The proposed bipolar myoelectric sensor-enabled human-machine interface could contribute to various scenarios of physical human-robot interaction.

The detection of moving objects in videos is very important in many video processing applications, and background modeling is often an indispensable process to achieve this goal. Most of the traditional background modeling methods utilize color or texture information. However, color information is sensitive to illumination variations and texture information cannot be utilized to separate smooth foreground from smooth background in most cases. Achieving good performance in terms of high foreground detection accuracy and low computational cost is also challenging. In this paper, we propose a new integration framework of texture and color information for background modeling, in which the foreground decision equation includes three parts (one part for color information, one part for texture information, and the left part for the integration of color and texture information). This framework is able to combine the advantages of texture and color features while inhibiting their disadvantages as well. Moreover, we propose a block-based method to accelerate the background modeling. In particular, in the texture information modeling process, a single histogram model is established for each block whose bins indicate the occurrence probabilities of different patterns, which is different from the traditional multihistogram model for block-based background modeling, and then dominant background patterns are selected to calculate the background likelihood of new coming blocks. Dynamic background and multimodal problems can be handled through this technique. To evaluate the foreground detection performance reasonably, a new quality measure is proposed. Extensive experiments on various challenging videos validate the effectiveness of the proposed method over state-of-the-art methods.

Efficient algorithms for mining frequent itemsets are crucial for mining association rules as well as for many other data mining tasks. Methods for mining frequent itemsets have been implemented using a prefix-tree structure, known as an FP-tree, for storing compressed information about frequent itemsets. Numerous experimental results have demonstrated that these algorithms perform extremely well. In this paper, we present a novel FP-array technique that greatly reduces the need to traverse FP-trees, thus obtaining significantly improved performance for FP-tree-based algorithms. Our technique works especially well for sparse data sets. Furthermore, we present new algorithms for mining all, maximal, and closed frequent itemsets. Our algorithms use the FP-tree data structure in combination with the FP-array technique efficiently and incorporate various optimization techniques. We also present experimental results comparing our methods with existing algorithms. The results show that our methods are the fastest for many cases. Even though the algorithms consume much memory when the data sets are sparse, they are still the fastest ones when the minimum support is low. Moreover, they are always among the fastest algorithms and consume less memory than other methods when the data sets are dense.

Mining frequent itemsets is at the core of mining association rules, and is by now quite well understood algorithmically for main memory databases. In this paper, we investigate approaches to mining frequent itemsets when the database or the data structures used in the mining are too large to fit in main memory. Experimental results show that our techniques reduce the required disk accesses by orders of magnitude, and enable truly scalable data mining.

Keys are very important in many aspects of data management, such as guiding query formulation, query optimization, indexing, etc. We consider the situation where an XML document does not come with key definitions, and we are interested in using data mining techniques to obtain a representation of the keys holding in a document. In order to have a compact representation of the set of keys holding in a document, we define a partial order on the set of all key expressions. This order is based on an analysis of the properties of absolute and relative keys for XML. Given the existence of the partial order, only a reduced set of key expressions need to be discovered.Due to the semistructured nature of XML documents, it turns out to be useful to consider keys that hold in "almost" the whole document, that is, they are violated only in a small part of the document. To this end, the support and confidence of a key expression are also defined, and the concept of approximate key expression is introduced. We give an efficient algorithm to mine a reduced set of approximate keys from an XML document.