In E-learning, accurately recognizing the learners’ emotions is a crucial prerequisite for enhancing learning outcomes and teaching quality. Most existing emotion recognition studies identify the emotions of learners by integrating their physiological signals and facial expressions, but these studies often overlook the impact of the different hierarchy of semantics embedded in instructional videos on learners’ emotion. Therefore, we innovatively propose an Emotion Recognition Model based on Multimodal Video Semantic Hierarchy. This model constructs hierarchical video semantics and gradually integrates them through hierarchical stacking. This fused semantic representation is then combined with the learners’ eye-movement physiological signals to enhance emotion recognition performance. Experimental results on three public multimodal physiological datasets, VLMED, HCI-Tagging and DEAP, confirms the model’s effectiveness in emotion recognition tasks.

Recent progress in recognizing emotions through gait analysis has attracted substantial interest. Spatial temporal graph convolutional networks (ST-GCN) have been applied to extract gait features effectively, enabling enhanced emotion recognition. However, existing methods do not account for subtle movement cues that are intricately linked to human emotions, resulting in a lack of representation of emotion intensity. Additionally, these methods fail to consider the cyclic nature of gait, focusing only on local dependencies in the temporal domain. To tackle these limitations, we propose an innovative three-streams graph neural network model PMF-GCN (Posture-Movement-Frequency-enhanced Graph Convolutional Network). First, we introduce for the first time in gait emotion recognition, the integration of movement features from translational and rotational perspectives, combined with frequency-domain data, to capture emotion intensity and global dependencies. Then, we devise a novel adaptive feature fusion mechanism (TR-AFM), which achieves effective extraction of spatial-temporal-specific emotion features from gaits through temporal and spatial attention mechanisms, as well as gated units. Comprehensive experiments on two public datasets show that PMF-GCN achieves leading performance in gait emotion recognition, and achieves state-of-the-art performance.

Person image synthesis with controllable body poses and appearances is an essential task owing to the practical needs in the context of virtual try-on, image editing and video production. However, existing methods face significant challenges with details missing, limbs distortion and the garment style deviation. To address these issues, we propose a Disentangled Representations Diffusion Model (DRDM) to generate photorealistic images from source portraits in specific desired poses and appearances. First, a pose encoder is responsible for encoding pose features into a high-dimensional space to guide the generation of person images. Second, a body-part subspace decoupling block (BSDB) disentangles features from the different body parts of a source figure and feeds them to the various layers of the noise prediction block, thereby supplying the network with rich disentangled features for generating a realistic target image. Moreover, during inference, we develop a parsing map-based disentangled classifier-free guided sampling method, which amplifies the conditional signals of texture and pose. Extensive experimental results on the Deepfashion dataset demonstrate the effectiveness of our approach in achieving pose transfer and appearance control. The associated project can be found at https://github.com/lovemusiceb/DRDM

Traffic flow prediction forms a critical foundation for the management and planning of urban transportation systems. However, the complex spatial interactions among road segments and the dynamic patterns of traffic flow variations across multiple time scales pose significant challenges to improving forecasting accuracy. To address these complexities, this paper introduces a Diversified Spatio-Temporal Feature Extraction (DSTF) Model, designed to effectively mine temporal, spatial, and spatio-temporal cross-correlations. Specifically, in the temporal dimension, a gated convolution-enhanced Res2Net architecture is employed to capture diverse traffic flow patterns across varying time scales. In the spatial dimension, the model integrates global and local perspectives by employing spatial attention mechanisms and a dual-view Geom-GCN, enabling it to capture global spatial dependencies, local geographic neighborhood relationships, and semantic similarity-driven spatial correlations among nodes within the urban road network. For spatio-temporal cross-correlations, a dynamic synchronization aggregation module is developed using spatio-temporal attention, effectively capturing the evolving interactions between nodes or regions across different time slices. Experimental evaluations conducted on four real-world highway traffic datasets demonstrate that DSTF outperforms PDFormer, achieving average improvements of 1.35%, 1.61% and 0.81% in MAE, MAPE and RMSE metrics, respectively. These results highlight the model’s superior capability in extracting and leveraging temporal and spatial features for traffic flow prediction. • Addresses issues related to dynamic traffic flow data. • Captures multi-scale time features of traffic flow temporally. • Models the dynamism of traffic flow in both temporal and spatial dimensions. • Surpasses state-of-the-art methods.

Conversational recommendation is one system that can extract the user's preferences and recommend suitable items in a similar way to human-like responses. Existing methods often use the feature extraction combined with the Transformer model to extract user preferences and make recommendations. However, these methods have two limitations. First, they do not consider the order in which entities appear, thus affecting the extraction of user preferences. Second, the generated responses lack diversity that affects the users’ experience to the system. To this end, we propose a conversational recommendation model with User Entity focus and Multi-Granularity latent variable enhancement (UEMG). In UEMG, we design a novel neural network that utilizes Bi-GRU to capture the appearing orders of entities in dialogues, and leverages Transformer to capture the global dependencies of entities, and then combines them to extract user preferences. For the second issue, to improve the diversity of dialogue generation, we propose a multi-granularity latent variable mechanism, which can extract more entities from the context information and the knowledge graphs, respectively. We conducted extensive experiments on publicly available dialogue generation datasets. Experimental results demonstrate that compared to current state-of-the-art methods, UEMG achieves 9.7% improvements in recommendation performance and 23% improvements in dialogue generation.

Multi-view learning is a machine learning paradigm that utilizes multiple feature sets or data sources to improve learning performance and generalization. However, existing multi-view learning methods often do not capture and utilize information from different views very well, especially when the relationships between views are complex and of varying quality. In this paper, we propose a novel multi-view learning framework for the multi-view classification task, called Gated Cross-Correlation Network (GCCNet), which addresses these challenges by integrating the three key operational levels in multi-view learning: representation, fusion, and decision. Specifically, GCCNet contains a novel component called the Multi-View Gated Information Distributor (MVGID) to enhance noise filtering and optimize the retention of critical information. In addition, GCCNet uses cross-correlation analysis to reveal dependencies and interactions between different views, as well as integrates an adaptive weighted joint decision strategy to mitigate the interference of low-quality views. Thus, GCCNet can not only comprehensively capture and utilize information from different views, but also facilitate information exchange and synergy between views, ultimately improving the overall performance of the model. Extensive experimental results on ten benchmark datasets show GCCNet's outperforms state-of-the-art methods on eight out of ten datasets, validating its effectiveness and superiority in multi-view learning.

Role-oriented dialogue summarization aims at generating summaries for different roles in dialogue, e.g., user and agent. Interaction between different roles is vital for the task. Existing methods could not fully capture interaction patterns between roles when encoding dialogue, thus are prone to ignore the interaction-related key information. In this paper, we propose a contrastive learning based interaction-aware model for the role-oriented dialogue summarization namely CIAM. An interaction-aware contrastive objective is constructed to guide the encoded dialogue representation to learn role-level interaction. The representation is then used by the decoder to generate role-oriented summaries. The contrastive objective is trained jointly with the primary dialogue summarization task. Additionally, we innovatively utilize different decoder start tokens to control what kind of summary to generate, thus could generate different role-oriented summaries with a unified model. Experimental results show that our method achieves new state-of-the-art results on two public datasets. Extensive analyses further demonstrate that our method excels at capturing interaction information between different roles and producing informative summaries.

Emotion Recognition in Conversations (ERC) plays a crucial role in the development of human-machine interaction. Conversations are a multi-party, multi-emotion, and multi-turn process of information propagation. However, existing works, which focus on designing models and algorithms for better learning representations of dialogue context and speakers, but rarely care about the key element of the strong correlation and inseparable interdependence of emotional states on the sentiment polarity in the process. To address this issue, we propose a novel model, named S2D-ERC (Sentiment-Speaker-Dependency for Emotion Recognition in Conversation), for ERC task. The proposed model constructs a conversation as a directed acyclic graph and represents both speaker- and sentiment- dependencies between utterances with heterogeneous edges. Additionally, to capture the information interaction dynamics in conversation context, we employ a cross-attention mechanism where latent representations of speaker and sentiment are learned with two different directions of information flow. The experimental results on two benchmarks, compared with state-of-the-art models, demonstrate the superiority and effectiveness of our model.

DailyConnect is a visual-based mobile application that supports children with autism spectrum disorder (ASD) in recalling memories by reviewing photos through discrete trial training (DTT) to understand situation-based emotions. To assess DailyConnect and its adaptability to a child’s characteristics and emotional situations, a pilot study was conducted that included 15 children with ASD and their parents and teachers. The DTT steps—memory recall and situation recognition, emotion recognition, emotion cues, facial expression recognition, and response behavior—were reliable in assessing the understanding of emotional situations when compared before and after the intervention in four categories of emotional situations, namely, happiness, sadness, anger, and fear. The results revealed that DailyConnect improves the understanding of situation-based emotions, particularly negative emotions (e.g., sadness: mean diff = −.687, sig. < .01, T = −3.866, d = 1.006; anger: mean diff = −.952, sig. < .01, T = −6.187, d = .705; fear: mean diff = −.961, sig. < .01, T = −5.522, d = .627); however, its effectiveness varied for children in different emotional situations. Furthermore, subjective feedback from participants and users (parents and teachers) provided insights into design considerations for similar mobile aids.

In realistic conversations, “responses” are closely related to persona and context. However, current personalized dialogue generation methods only focus on the consistency of the persona of responses and yet ignore context coherence, as may produce low-quality responses. To address the issue, we propose a novel model, named PCF (Persona and Context Fusion), which builds two decoders for understanding personality consistency and context coherence respectively on a common encoder-decoder architecture. In this model, an inter-layer attention fusion mechanism is designed for the two decoders to effectively fuse persona and context, and a decoupled way of training is conducted on an additional large-scale non-dialogue inference dataset to enhance the consistent understanding ability of the two decoders. Furthermore, considering that generating responses may be tedious, the ScaleGrad loss function is applied to enhance the diversity of responses. Experimental results on two publicly available datasets show that the dialogues generated by our PCF are significantly higher in quality than strong baselines.

Mobile and Web-of-Things (WoT) devices at the network edge account for more than half of the world’s web traffic, making a great data source for various machine learning (ML) applications, particularly federated learning (FL) which offers a promising solution to privacy-preserving ML feeding on these data. FL allows edge mobile and WoT devices to train a shared global ML model under the orchestration of a central parameter server. In the real world, due to resource heterogeneity, these edge devices often train different versions of models (e.g., VGG-16 and VGG-19) or different ML models (e.g., VGG and ResNet) for the same ML task (e.g., computer vision and speech recognition). Existing FL schemes have assumed that participating edge devices share a common model architecture, and thus cannot facilitate FL across edge devices with heterogeneous ML model architectures. We explored this architecture heterogeneity challenge and found that FL can and should accommodate these edge devices to improve model accuracy and accelerate model training. This paper presents our findings and FlexiFed, a novel scheme for FL across edge devices with heterogeneous model architectures, and three model aggregation strategies for accommodating architecture heterogeneity under FlexiFed. Experiments with four widely-used ML models on four public datasets demonstrate 1) the usefulness of FlexiFed; and 2) that compared with the state-of-the-art FL scheme, FlexiFed improves model accuracy by 2.6%-9.7% and accelerates model convergence by 1.24 × -4.04 ×.

Lane detection is critical for intelligent vehicles to sense drivable areas. Compared to general objects, lane lines are slender-shaped, easily occluded, or defaced. Therefore, the lane detection network requires a more robust ability for local detail extraction and global semantic information modeling. In this paper, we propose a novel lane detection network (FLAMNet) with a flexible line anchor mechanism, which constantly corrects the position of line anchors to improve detection performance and computational efficiency. Specifically, we utilize the Patch Pooling Aggregation Module (PPAM) to aggregate multi-scale semantic features extracted by the backbone network. The multi-scale features are subsequently inputted into DSAformer, which utilizes decomposed self-attention to establish global long-distance dependencies. The detection head leverages fused features of multi-scale global and local details to accurately fit the lane line by correcting the anchor position. Moreover, we propose the Horizontal Information Aggregation Module (HIAM) to expand the receptive field of line anchors horizontally, enhancing the line anchor representation ability to the topological structure of complex lane lines. The experimental results on mainstream lane detection benchmark datasets demonstrate that the proposed FLAMNet outperforms existing methods. We have uploaded the code and demo of FLAMNet on GitHub at:https://github.com/RanHao-cq/FLAMNet.

In the recommendation system, click-through rate (CTR) prediction is a popular research direction. Aiming at the problem of excessive compression of features in Factorization Machine (FM) and its variant models, a recommendation model that combines feature interaction enhancement and multi-head self-attention is proposed. Hadamard product, feature vector splicing and multi-layer perception network methods are used for low-level feature vector interactive processing in this paper, and multi-head self-attention mechanism and residual network model for high-level feature interactive processing are used. By designing the fusion mechanism, the parallel low-order feature interaction network and the high-order feature interaction network are merged. The experimental results on the four benchmark data sets show that the multi-head self-attention model based on high-order feature interaction enhancement proposed in this paper outperforms existing models in terms of click-through rate prediction accuracy.

Conversational recommender system (CRS) allows agent to understand the conversation with user and give recommendations after multi-turn dialogues. However, there are still two limitations in existing CRS: (1) improper words or items may be chosen for the given topic in the generation, and (2) the contextual information of items in the recommendation is not rationally explored. To solve these issues, we proposed a Keyword-guided Topic-oriented CRS model (KGTO), which captures more accurate topic by extracting keywords through the hierarchical attention mechanism, and enriches the contextual information of items by fusing the co-occurrence graph with the knowledge graph. Moreover, a generative module can select words or items supplemented topic information to generate proper responses. Extensive experiments on the task-oriented dialogue dataset prove that our model performs well in recommendation effectiveness and dialogue informativeness.

Dialogue has received a lot of research. But there is very little research on dialogue sentiment forecasting, which aims to forecast the sentimental polarity of what the interlocutor is about to say and provides sentimental guidance for empathic dialogue generation. Since the sentence has not been spoken, the vector of the sentence can't be directly obtained. And according to cognitive science, emotions are different from sentiment, but there is an internal connection. Therefore, our paper proposes an Emotion-Assisted Sentiment Forecasting (EASF) model based on attention to integrating these goals. Our model uses attention to capture the significant content of emotions and sentiment, and emotion assistance can obtain the emotional change, then this change is used to assist in the analysis of the polarity of the sentiment. Experimental results show that EASF significantly outperforms all baselines.

Recent advances in localization techniques have fundamentally enhanced social networking services, allowing users to share their locations and location-related contents. This has further increased the popularity of location-based social networks (LBSNs) and produces a huge amount of trajectories composed of continuous and complex spatio-temporal points from people’s daily lives. How to accurately aggregate large-scale trajectories is an important and challenging task. Conventional clustering algorithms (e.g., k -means or k -mediods) cannot be directly employed to process trajectory data due to their serialization, triviality and redundancy. Aiming to overcome the drawbacks of traditional k -means algorithm and k -mediods, including their sensitivity to the selection of the initial k value, the cluster centers and easy convergence to a locally optimal solution, we first propose an optimized k -means algorithm (namely OKM ) to obtain k optimal initial clustering centers based on the density of trajectory points. Second, because k -means is sensitive to noisy points, we propose an improved k -mediods algorithm called IKMD based on an acceptable radius r by considering users’ geographic location in LBSNs. The value of k can be calculated based on r , and the optimal k points are selected as the initial clustering centers with high densities to reduce the cost of distance calculation. Thirdly, we thoroughly analyze the advantages of IKMD by comparing it with the commonly used clustering approaches through illustrative examples. Last, we conduct extensive experiments to evaluate the performance of IKMD against seven clustering approaches including the proposed optimized k -means algorithm, k -mediods algorithm, traditional density-based k -mediods algorithm and the state-of-the-arts trajectory clustering methods. The results demonstrate that IKMD significantly outperforms existing algorithms in the cost of distance calculation and the convergence speed. The methods proposed is proved to contribute to a larger effort targeted at advancing the study of intelligent trajectory data analytics.

Long short-term memory (LSTM) neural networks and attention mechanism have been widely used in sentiment representation learning and detection of texts. However, most of the existing deep learning models for text sentiment analysis ignore emotion's modulation effect on sentiment feature extraction, and the attention mechanisms of these deep neural network architectures are based on word- or sentence-level abstractions. Ignoring higher level abstractions may pose a negative effect on learning text sentiment features and further degrade sentiment classification performance. To address this issue, in this article, a novel model named AEC-LSTM is proposed for text sentiment detection, which aims to improve the LSTM network by integrating emotional intelligence (EI) and attention mechanism. Specifically, an emotion-enhanced LSTM, named ELSTM, is first devised by utilizing EI to improve the feature learning ability of LSTM networks, which accomplishes its emotion modulation of learning system via the proposed emotion modulator and emotion estimator. In order to better capture various structure patterns in text sequence, ELSTM is further integrated with other operations, including convolution, pooling, and concatenation. Then, topic-level attention mechanism is proposed to adaptively adjust the weight of text hidden representation. With the introduction of EI and attention mechanism, sentiment representation and classification can be more effectively achieved by utilizing sentiment semantic information hidden in text topic and context. Experiments on real-world data sets show that our approach can improve sentiment classification performance effectively and outperform state-of-the-art deep learning-based methods significantly.

Recommender systems based on collaborative filtering has always suffered from sparsity and cold start problems. Therefore, researchers attempt to address the issues with various side information such as user profiles and item attributes. In this paper, we proposed Graph-aware Collaborative Filtering (GCF), an end-to-end framework, in which user-item bipartite graph and the knowledge graph of items (side information) are integrated to improve recommendation performance. In GCF, we aggregate the neighbors in the candidate item knowledge graph to refine the item representation. Similarly, we aggregate the user interaction neighbors in the user-item bipartite graph to refine the user representation. The collaborative signals in the knowledge graph and user-item bipartite graph are successfully captured through the neighborhood aggregation operation. Experimental results on three real datasets indicate that the proposed GCF is superior to the existing models in terms of accuracy and can also effectively solve the data sparsity problem of the recommender system.

In view of the problems of supervised machine learning methods, such as low accuracy, long training time, complicated models, and poor versatility, this paper proposes a method for discovering ECG patterns based on local repeatability. By using the sliding window technique, the patterns contained in the ECG time series data are mined; by using the pattern matching technique, a method for improving the similarity of the ECG time series is explored. The paper implements an ECG pattern discovery algorithm based on local repeatability, accurately calculates the similarity between two ECG patterns, and performs clustering and labeling based on these similarities. Experimental results show that the method proposed in this paper is superior to the existing methods in terms of accuracy of pattern discovery and stability of pattern discovery.

The choice of image feature representation plays a crucial role in the analysis of visual information. Although vast numbers of alternative robust feature representation models have been proposed to improve the performance of different visual tasks, most existing feature representations [e.g., handcrafted features or convolutional neural networks (CNNs)] have a relatively limited capacity to capture the highly orientation-invariant (rotation/reversal) features. The net consequence is suboptimal visual performance. To address these problems, this study adopts a novel transformational approach, which investigates the potential of using polar feature representations. Our low level consists of a histogram of oriented gradient, which is then binned using annular spatial bin-type cells applied to the polar gradient. This gives gradient binning invariance for feature extraction. In this way, the descriptors have significantly enhanced orientation-invariant capabilities. The proposed feature representation, calledorientation-invariant histograms of oriented gradients, is capable of accurately processing visual tasks (e.g., facial expression recognition). In the context of the CNN architecture, we propose two polar convolution operations, referred to as full polar convolution and local polar convolution, and use these to develop polar architectures for the CNN orientation-invariant representation. Experimental results show that the proposed orientation-invariant image representation, based on polar models for both handcrafted features and deep learning features, is both competitive with state-of-the-art methods and maintains compact representation on a set of challenging benchmark image datasets.

To automatically determine the number of clusters and generate more quality clusters while clustering data samples, we propose a harmonious genetic clustering algorithm, named HGCA, which is based on harmonious mating in eugenic theory. Different from extant genetic clustering methods that only use fitness, HGCA aims to select the most suitable mate for each chromosome and takes into account chromosomes gender, age, and fitness when computing mating attractiveness. To avoid illegal mating, we design three mating prohibition schemes, i.e., no mating prohibition, mating prohibition based on lineal relativeness, and mating prohibition based on collateral relativeness, and three mating strategies, i.e., greedy eugenics-based mating strategy, eugenics-based mating strategy based on weighted bipartite matching, and eugenics-based mating strategy based on unweighted bipartite matching, for harmonious mating. In particular, a novel single-point crossover operator called variable-length-and-gender-balance crossover is devised to probabilistically guarantee the balance between population gender ratio and dynamics of chromosome lengths. We evaluate the proposed approach on real-life and artificial datasets, and the results show that our algorithm outperforms existing genetic clustering methods in terms of robustness, efficiency, and effectiveness.

Finding overlapping communities from multimedia social networks is an interesting and important problem in data mining and recommender systems. However, extant overlapping community discovery with swarm intelligence often generates overlapping community structures with superfluous small communities. To deal with the problem, in this paper, an efficient algorithm (LEPSO) is proposed for overlapping communities discovery, which is based on line graph theory, ensemble learning, and particle swarm optimization (PSO). Specifically, a discrete PSO, consisting of an encoding scheme with ordered neighbors and a particle updating strategy with ensemble clustering, is devised for improving the optimization ability to search communities hidden in social networks. Then, a postprocessing strategy is presented for merging the finer-grained and suboptimal overlapping communities. Experiments on some real-world and synthetic datasets show that our approach is superior in terms of robustness, effectiveness, and automatically determination of the number of clusters, which can discover overlapping communities that have better quality than those computed by state-of-the-art algorithms for overlapping communities detection.