Large Language Models (LLMs) have demonstrated remarkable proficiency in diverse tasks. This success raises a fundamental question in machine composition: Can symbolic music be considered a special form of language that can be jointly modeled with natural language for composition tasks? Recent studies validate that symbolic music can be modeled as a human language, yet composing structured music from partial symbolic inputs through natural language interaction remains underexplored. Even LLMs struggle to generate structurally coherent compositions in such hybrid input-output scenarios, highlighting a fundamental gap that calls for a domain-specific learning paradigm. To this end, we propose Inspiration-to-Structure (IoS), a cognitively inspired framework that enables LLMs to generate structured musical sections from melodic ideas. IoS employs a three-phase process—semantic, structural, and collaborative cognition—and is supported by two key components: (1) a new dataset and construction protocol called Structured Triplet Data (STD), and (2) a training method, Dual-Instance Structural Contrastive Optimization (DiSCO), designed to enhance structural awareness. Experiments show that IoS improves structural coherence by 47.8% and artistic creativity by 21.8% compared to conventional language modeling paradigm, supervised fine-tuning, and even enables smaller LLMs to surpass larger LLMs. These results suggest that symbolic music, while language-like, demands specialized modeling beyond standard language modeling paradigms. IoS enables LLMs to transform music theory knowledge into structured composition, empowering users to compose music interactively via language and advancing toward general creative AI.

Recent advances in Referring Expression Comprehension (REC) have been largely driven by supervised learning on curated datasets, where each expression is assumed to refer to exactly one known object. However, such assumptions rarely hold in real-world scenarios, where expressions can refer to multiple objects, fail to refer to any, or involve novel categories and complex semantics. These challenges define the task of open-world REC, which demands robust generalization and structured reasoning beyond the scope of traditional REC methods. In this work, we introduce a novel, training-free framework that decouples visual perception from linguistic reasoning to address open-world REC. Our method first transforms the visual scene into a rich textual representation using an open-vocabulary multimodal perception module. It then employs a reasoning language model to interpret the referring expression and perform explicit logical inference over the perceived scene, enabling transparent decision-making and strong generalization in open-world scenarios. Experiments on three standard REC benchmarks as well as two more challenging ones, gRefCOCO and D³, demonstrate that our framework achieves highly competitive zero-shot performance, often surpassing supervised baselines.

Motor imagery-based brain-computer interfaces have been increasingly applied in post-stroke motor rehabilitation. Their effectiveness depends on accurate electroencephalography (EEG)-based motor imagery (MI) classification to provide reliable feedback. However, differences in MI-related EEG patterns between left- and right-sided hemiparesis result in heterogeneity among stroke patients, complicating MI classification. Most classification models are developed for healthy subjects and fail to capture stroke-specific neurophysiology. To address this, we propose HSAMoE, a hemiparetic-side-aware mixture of experts framework. HSAMoE employs a router to identify the hemiparetic side and directs data to the corresponding specialized expert, which specifically models the non-lesioned hemisphere where MI patterns are reliably preserved. Experiments on the Stroke MI 50 and Stroke19-HS datasets demonstrate that HSAMoE achieves superior performance over competing methods.

Speech Referring Expression Comprehension (SREC) aims to localize the object in an image referred to by a spoken natural language query. However, raw speech is continuous and noisy, and prior ASR-free methods that align full utterances with transcripts using only global supervision can overfit to spurious correlations, limiting fine-grained grounding. To address this issue, we propose SliceCSRef, a robust SREC framework that improves generalization via dual-level semantic alignment. Beyond utterance-level speech–text alignment, SliceCSRef introduces slice-wise alignment that pairs randomly sampled speech segments with weakly matched transcript spans based on their relative temporal positions, providing fine-grained supervision without additional annotations. Experiments on six benchmarks show that SliceCSRef achieves state-of-the-art performance under standard settings and consistently improves robustness under truncated speech and playback-speed variations.

Compared to conventional image content analysis tasks, visual emotion analysis is perceived as a complex, abstract, and potentially culturally dependent endeavor. The accuracy of automatic image-based emotion recognition remains a challenge, and the most significant obstacles are the affective gap and scarcity of data, particularly labeled data. To address these challenge, this paper proposes a saliency-guided masked image modeling approach. Specifically, the proposed framework employs multi-modal large model to generate more emotional images, thereby reducing the impact of data scarcity on model performance. Subsequently, neuroimaging and behavioral studies have demonstrated that human visual attention is attracted by the emotional relevance of a stimulus. In light of this, our model employs a saliency-guided masking strategy to identify emotion-related regions for masking sampling to fit the affective gap. In contrast to the conventional approach of using the original pixel values for the reconstruction target, our model eliminates high-frequency components from the pixels, thus enhancing the generalizability of the model. The use of this unsupervised representation learning approach enables the model to exhibit outstanding recognition performance in downstream emotion recognition tasks on three standard emotion datasets. Furthermore, ablation experiments, robustness test, and visualization experiments corroborate the effectiveness of the proposed method.

Electroencephalography (EEG)-based affective computing has achieved significant progress due to the rapidly developing learning model. However, compared to computer vision or natural language processing-related tasks, EEG data presents challenges such as a low signal-to-noise ratio, small sample size, and nonstationary properties. These factors may affect the model’s performance on cross-domain tasks, suggesting that further improvements are necessary. Existing models aim to minimize intraclass distance and maximize interclass distance to achieve an optimal solution. This can potentially have a negative impact on the model’s generalization ability because they ignore the existence of low-quality training samples that may not be suitable for strict optimization. This article presents a method for determining sample quality based on the guidance of knowledge from emotional neuroscience. The method differentiates between high- and low-quality training samples and designs a corresponding loss function to impose intraclass and interclass constraints on hyperspherical manifolds based on the quality of the samples. Therefore, it can be argued that our proposed method achieves a better balance between reducing the intraclass distance of high-quality samples and preventing the overfitting of low-quality ones. This could potentially help build a more robust EEG emotion recognition model. Numerous experiments are conducted on the SJTU Emotion EEG Dataset (SEED) and SJTU Emotion EEG Dataset (IV) (SEED-IV) datasets under cross-subject and cross-session scenarios, which show the superior performance of our proposed method and its higher recognition accuracy against adversarial attacks.

The accumulating body of research indicates that Motor Imagery (MI)-BCIs have the potential to enhance the quality of life for individuals with disabilities and to advance our understanding of brain function and rehabilitation strategies. Among these diseases, stroke is the leading cause of long-term motor disability across the globe, thereby underscoring the need for innovative rehabilitation strategies, such as MI-BCI technologies. In contrast with these expectations, the majority of existing research is built upon data obtained from healthy subjects. The construction of effective classification models for Motor Imagery tasks in patients with brain diseases, particularly stroke, remains a significant challenge. The lateralization of the left and right hemispheres is more pronounced in patients who have suffered a stroke than in healthy individuals. Moreover, the specific locations of lesions and the regions of influence result in significant variations in the electroencephalogram (EEG) data of patients with different hemiplegic sides. This paper explores the potential of generative models in addressing the issue of domain differences arising from different hemiplegic sides EEG data. Furthermore, this paper circumvents the potential adverse effects of rigorous optimization of low-quality samples on model performance through the utilization of label softening algorithm. Two MI-EEG datasets of stroke patients performing Motor Imagery tasks are used to validate our method. In comparison to both classical machine learning methods and those state-of-the-art models for MI classification, the classification model in this paper achieves a noticeable performance improvement in different data partitioning strategies, including subject-dependent and subject-independent scenarios. Each sub-module, and each designed loss function, contributes to the final performance growth. In addition, this paper also investigates the potential of the proposed framework for predicting a patient's level of functional recovery. Our findings indicate that the addition of a prediction layer to the proposed model enables the accurate prediction of functional recovery level in stroke patients.

Literature reviews play a crucial role in scientific research for understanding the current state of research, identifying gaps, and guiding future studies on specific topics. However, the process of conducting a comprehensive literature review is yet time-consuming. This paper proposes a novel framework, collaborative knowledge minigraph agents (CKMAs), to automate scholarly literature reviews. A novel prompt-based algorithm, the knowledge minigraph construction agent (KMCA), is designed to identify relations between concepts from academic literature and automatically constructs knowledge minigraphs. By leveraging the capabilities of large language models on constructed knowledge minigraphs, the multiple path summarization agent (MPSA) efficiently organizes concepts and relations from different viewpoints to generate literature review paragraphs. We evaluate CKMAs on three benchmark datasets. Experimental results show the effectiveness of the proposed method, further revealing promising applications of LLMs in scientific research.

Motor imagery-based brain-computer interfaces (MI-BCIs) require abundant high-quality EEG data, yet collecting such data remains challenging. Recent diffusion probabilistic models (DPMs) have shown strong capabilities in generating realistic data, yet synthesizing EEG signals that preserve neuroscientific relevance remains challenging. To address this issue, we propose DiffEEGLossNet, a diffusion-based generative framework tailored for multi-channel EEG synthesis. By introducing an ERD/ERS-informed loss function, the model constrains the diffusion process to maintain motor imagery-specific neural patterns, enhancing both physiological plausibility and data fidelity. Experiments on the BCI Competition IV 2a and 2b datasets demonstrate that the generated EEG signals not only exhibit high realism and consistency but also significantly boost the performance of downstream EEG classification models through effective data augmentation.

The cross-task decoding capability of electroencephalography (EEG) is of significant importance for the practical application of Brain-Computer Interfaces (BCIs). However, the issue of feature drift in EEG signals severely limits the generalization ability of decoding algorithms within BCIs. Traditional methods struggle to effectively model stable neural representations across different cognitive tasks and lack constraints on the brain function-structure coupling relationship. To address this, this paper proposes a cross-task decoding framework (PGN) that integrates prototype learning with graph neural networks. The framework captures task-invariant features by constructing a hierarchical prototype network and employs automated brain parcellation to constrain dynamic local graph learning. Additionally, intra-class aggregation and inter-class separation losses are combined to reduce distribution discrepancies across tasks. The proposed unified model can handle three types of tasks: emotion, fatigue, and cognitive load. Experiments on multiple standard cognitive task datasets, including SEED-VIG, DEAP, and memory load, demonstrate that our method significantly outperforms traditional baseline models. Notably, it achieves an accuracy of 85.51 % on the SEED-VIG fatigue monitoring task, reaching the state-of-the-art (SOTA) performance. Our code will be made public on github.

Cognitive diagnosis (CD) aims to identify the extent to which learners have mastered different knowledge concepts, which is an essential task for online education platforms. Due to the increase in educational data and advancements in deep learning technologies, numerous CD models have been proposed, achieving promising results. However, most existing models lack a comprehensive understanding of learners’ behavioral patterns, focusing primarily on global knowledge structures while overlooking fine-grained response behaviors. Therefore, we propose LAB-ICDM, an Inductive Cognitive Diagnosis Model Integrating the Characteristics of Learners’ Answering Behavior. Specifically, (i) we introduce a global relation aggregation module, which constructs a learner-centered graph to capture global representations of learners; (ii) we design a relation-aware module, which leverages a learner-exercise graph to extract fine-grained behavioral patterns, enabling a more comprehensive representation of learners’ exercise-solving behaviors and their interactions with exercises. By fusing these two modules, our model constructs a multi-view representation of learners’ knowledge states, enabling more accurate cognitive diagnosis. Extensive experiments on three real-world educational datasets demonstrate that LAB-ICDM significantly outperforms existing state-of-the-art models, showcasing superior predictive performance.

Word Sense Disambiguation (WSD) aims to determine the correct meaning of a word in context from a predefined inventory, and remains a fundamental challenge in natural language understanding.Existing methods rely heavily on manually annotated data, which limits coverage and generalization.In this work, we propose a scalable framework that leverages large language models (LLMs) as knowledge distillers to construct silver-standard WSD corpora.We explore generation-based distillation, where diverse examples are synthesized for dictionary senses, and annotation-based distillation, where LLMs assign sense labels to polysemous words within real-world corpus sentences.The resulting data is used to train tiny models.Extensive experiments show that models distilled from LLM-generated data outperform those trained on gold-standard corpora, especially on general-domain benchmarks.Our annotationbased model, after balancing sense distribution, achieves 50% F1 gain on the most challenging test set and the best distilled model can match or even exceed the performance of its LLM teacher, despite having over 1000 times fewer parameters.These results demonstrate the effectiveness of LLM-based distillation for building accurate, generalizable, and efficient WSD systems.

With the widespread adoption of cloud computing and containerization technologies, large-scale clusters now host increasingly complex applications, yet the fragmentation of multidimensional resources (CPU, GPU, memory) has significantly degraded resource utilization efficiency. To address this challenge, this study investigates scheduling strategies for multidimensional resource fragmentation optimization. First, a unified quantitative metric is established by extending a task-statistics-based fragmentation measurement approach to threedimensional resource constraints (CPU, memory, GPU), enabling objective assessment of fragmentation levels. Subsequently, a multidimensional resource fragmentation optimization scheduling strategy is proposed based on the FGD framework. This approach employs TOPSIS to holistically address multidimensional resource fragmentation optimization. Building upon this methodology, we design and implement both two-dimensional and three-dimensional scheduling strategies. Comprehensive evaluation using an extended Kubernetes scheduler simulator with Alibaba production dataset reveals that: 1) All TOPSIS-based multi-dimensional strategies achieve performance comparable to or exceeding baseline approaches in their target dimensions while demonstrating synergistic effects.; 2) Two-dimensional strategies achieve optimal performance in their target dimensions but may degrade others 3) The three-dimensional optimization strategy TOPSIS-FGD-CGM can overcome the limitations of two-dimensional optimization strategies and demonstrates robust comprehensive performance.

Deep neural networks (DNNs) have become valuable assets due to their success in various tasks, but their high training costs also make them targets for model theft. Fingerprinting techniques are commonly used to verify model ownership, but existing methods either require training many additional models, leading to increased costs, or rely on GANs to generate fingerprints near decision boundaries, which may compromise image quality. To address these challenges, we propose a GAN-based fingerprint generation method that applies frequency-domain perturbations to normal samples, effectively creating fingerprints. This approach not only resists intellectual property (IP) threats, but also improves fingerprint acquisition efficiency while maintaining high imperceptibility. Extensive experiments demonstrate that our method achieves a state-of-the-art (SOTA) AUC of 0.98 on the Tiny-ImageNet dataset under IP removal attacks, outperforming existing methods by 8%, and consistently achieves the best ABP for three types of IP detection and erasure attacks on the GTSRB dataset. Our source code is available at https://github.com/wason981/Frequency-Fingerprinting.

Although deep learning models have shown promise in Electroencephalogram (EEG) - based emotion recognition, their precision and robustness are often limited by domain-specific bias and low signal-to-noise ratio of EEG. Among the challenges encountered, the disparities in EEG data between subjects have been observed to potentially exceed those observed between different states of the task. The integration of identity domain information with task domain information may cause the model's learning of subject-dependent information, thereby hindering the ability to discern differences between tasks. This paper proposes a novel deep learning framework that improves the performance of EEG-based emotion classification models through the removal of the identity domain. Specifically, the proposed framework utilizes Auto-encoder (AE) based model as denoising module to extract globally informative features from the EEG signals. The domain information extraction module extracts subject-specific features from EEG samples by minimizing the differences between the network's output and the subject's identity labels while ensuring consistency of intermediate features across samples from the same subject. By combining domain-specific and globally informative features, an additional domain removal network is introduced before the final classification network to mitigate domain-specific biases by stripping away identity domain information from globally informative features through pairwise contrastive learning. The proposed EEG-based emotion recognition framework, supported by extensive experimental evidence, shows that incorporating the suggested domain removal technique significantly improves the robustness of the classification model and achieves state-of-the-art results in emotion classification.

The high cost of developing high-performance deep models highlights their value as intellectual property for creators. However, it is important to consider the potential risks of theft. Although various techniques have been developed to protect the intellectual property of deep models, there is still room for improvement in terms of efficiency, comprehensiveness, and generalization. Compared with the intrusiveness of watermarking methods, fingerprinting methods do not affect the training process of the source model. Consequently, this paper proposes a fingerprinting method to address the paucity of attempts in fingerprinting methods for model protection. Our method consists of two efficient algorithms for generating fingerprinting samples, where the first one possesses the advantage of efficiency, while the second one is better in terms of robustness. The first algorithm takes a comprehensive approach to modeling the fingerprint of the deep model. The generated samples are distributed within the stable region and near the decision boundary of the model, taking into account both the duality and the conviction factors. Then, a heuristic sample perturbation algorithm is introduced, which generates a fingerprint with solid stability and generalization across multiple domains. The two algorithms proposed in this paper have been shown to be capable of withstanding attacks on intellectual property removal, detection, and evasion. They also show some advantages in terms of efficiency. In addition, the proposed method is the first to apply fingerprinting techniques in a cross-domain context.

Call-and-response is a musical technique that enriches the creativity of music, crafting coherent musical ideas that mirror the back-and-forth nature of human dialogue with distinct musical characteristics. Although this technique is integral to numerous musical compositions, it remains largely uncharted in automatic music composition. To enhance the creativity of machine-composed music, we first introduce the Call-Response Dataset (CRD) containing 19,155 annotated musical pairs and crafted comprehensive objective evaluation metrics for musical assessment. Then, we design a knowledge-enhanced learning-based method to bridge the gap between human and machine creativity. Specifically, we train the composition module using the call-response pairs, supplementing it with musical knowledge in terms of rhythm, melody, and harmony. Our experimental results underscore that our proposed model adeptly produces a wide variety of creative responses for various musical calls.

In recent years, using Electroencephalography (EEG) to recognize emotions has garnered considerable attention. Despite advancements, limited EEG data restricts its potential. Thus, Generative Adversarial Networks (GANs) are proposed to mimic the observed distributions and generate EEG data. However, for imbalanced datasets, GANs struggle to produce reliable augmentations for under-represented minority emotions by merely mimicking them. Thus, we introduce Emotional Subspace Constrained Generative Adversarial Networks (ESC-GAN) as an alternative to existing frameworks. We first propose the EEG editing paradigm, editing reference EEG signals from well-represented to under-represented emotional subspaces. Then, we introduce diversity-aware and boundary-aware losses to constrain the augmented subspace. Here, the diversity-aware loss encourages a diverse emotional subspace by enlarging the sample difference, while boundary-aware loss constrains the augmented subspace near the decision boundary where recognition models can be vulnerable. Experiments show ESC-GAN boosts emotion recognition performance on benchmark datasets, DEAP, AMIGOS, and SEED, while protecting against potential adversarial attacks. Finally, the proposed method opens new avenues for editing EEG signals under emotional subspace constraints, facilitating unbiased and secure EEG data augmentation.

The existing text-to-image generation methods based on stable diffusion yield better results in low-semantic prompt but often neglect the generation quality of high-semantic prompt such as emotional vocabulary, resulting in poor emotional image generation. In order to address this issue, we propose a novel approach called Sal-Guide Diffusion, which leverages saliency maps to guide emotional image generation with the goal of producing superior emotionally expressive images. In order to let the saliency maps guide the diffusion process, we introduce a lightweight adapter to extract emotional information from saliency maps and incorporate it into the diffusion process. Experimental results demonstrate that our proposed method generates higher-quality images across eight emotional dimensions, excelling in both generalization, emotional congruence, and subjective preference compared to stable diffusion or similar methods.

In the rapidly advancing field of deep learning, a significant yet often overlooked challenge is the protection of intellectual property (IP) for models based on electroencephalography (EEG). These models, which handle sensitive and private physiological information, have not received as much attention for IP protection as their counterparts in more mainstream areas like computer vision (CV) and natural language processing (NLP). This paper introduces an innovative fingerprinting method for the first time, targeting IP protection of EEG-based models, a domain where conventional watermarking techniques fall short. We design a novel conditional diffusion model, tailored to a universal EEG format, which is the first application of diffusion models in model IP protection. Furthermore, our retrieval strategy, characterized by three distinct conditions, facilitates the construction of the fingerprint validation set from synthesized EEG samples. Experiments demonstrate that our method not only outperforms existing state-of-the-art (SOTA) protection techniques in robustness against various IP attacks but also excels in generating high-quality and high-diversity EEG samples.

Deep neural networks (DNNs) are widely employed across various domains, with their training costs making them crucial assets for model owners.However, the rise of Machine Learning as a Service has made models more accessible, but also increases the risk of leakage.Attackers can successfully steal models through internal leaks or API access, emphasizing the critical importance of protecting intellectual property.Several watermarking methods have been proposed, embedding secret watermarks of model owners into models.However, watermarking requires tampering with the model's training process to embed the watermark, which may lead to a decrease in utility.Recently, some fingerprinting techniques have emerged to generate fingerprint samples near the classification boundary to detect pirated models.Nevertheless, these methods lack distance constraints and suffer from high training costs.To address these issues, we propose to utilize conditional generative network to generate fingerprint data points, enabling a better exploration of the model's decision boundary.By incorporating margin loss during GAN training, we can control the distance between generated data points and classification boundary to ensure the robustness and uniqueness of our method.Moreover, our method does not require additional training of proxy models, enhancing the efficiency of fingerprint acquisition.To validate the effectiveness of our approach, we evaluate it on CIFAR-10 and Tiny-ImageNet, considering three types of model extraction attacks, fine-tuning, pruning, and transfer learning attacks.The results demonstrate that our method achieves ARUC values of 0.186 and 0.153 on CIFAR-10 and Tiny-ImageNet datasets, respectively, representing a remarkable improvement of 400% and 380% compared to the current leading baseline.The source code is available at https://github.com/wason981/MarginFinger.

Knowledge Tracing (KT) focuses on estimating students' knowledge states and predicting their future performances, which is a crucial task for online education platforms. In light of the advancements in educational big data and deep neural networks, numerous KT models have been proposed and promising outcomes have been achieved. Nevertheless, we have noted that current methods possess certain evident constraints. Thus, we propose a Knowledge Tracing model with Multi-Relational Integration (MRIKT): (1) we consider the more sophisticated relations between questions and skills, which can reveal deeper patterns of students' learning; (2) we emphasize the forgetfulness nature of students and the value of inter-exercises relations by incorporating a retentive module. Specifically, we choose graph convolutional networks to construct the advanced-relation between questions and skills, named graph representation module. Additionally, by linking different exercises, our novel retentive module, inspired by RetNet, can acquire valuable insights. We extensively evaluate the performance of MRIKT on three real-world datasets. The results demonstrate that MRIKT achieves outstanding performance, which improves at least 8.44% compared to baseline models.

Explaining what part of the input images primarily contributed to the predicted classification results by deep models has been widely researched over the years and many effective methods have been reported in the literature, for which deep Taylor decomposition (DTD) served as the primary foundation due to its advantage in theoretical explanations brought in by Taylor expansion and approximation. Recent research, however, has shown that the root of Taylor decomposition could extend beyond local linearity, and thus causing DTD to fail in delivering expected performances. In this paper, we propose a universal root inference method to overcome the shortfall and strengthen the roles of DTD in explainability and interpretability of deep classifications. In comparison with the existing approaches, our proposed features in: (i) theoretical establishment of the relationship between ideal roots and the propagated relevances; (ii) exploitation of gradient descents in learning a universal root inference; and (iii) constrained optimization of its final root selection. Extensive experiments, including both quantitative and qualitative, validate that our proposed root inference is not only effective, but also delivers significantly improved performances in explaining a range of deep classifiers. We share our codes via the link: https://github.com/meetxinzhang/XAI-RootInference.

Most existing domain adaptation methods learn with both (labeled) samples in the source domain and (unlabeled) samples in the target domain. Relying on the availability of target domain samples, however, is not always feasible in real-world applications. In this paper, we propose a new method to address this issue, in which the target domain samples do not need to be available for the task of interest. To improve the performance of such a zero-shot domain adaptation (ZSDA), we learn with not only source samples in the task of interest, but also seek additional assistance from those dual-domain samples in an irrelevant task. To overcome the problems induced by the unavailability of target samples in the task of interest, we exploit the hypothesis that the domain correlation is consistent across tasks and learn to transfer it from the irrelevant task to the task of interest. Specifically, our method aims to learn a domain-invariant representation space in which the source-domain classifier is directly transferable to the target domain. We achieve this by restricting the two domains to share both inter-category structure and intra-category structure in the representation space. Experiment results on five benchmarking datasets indicate that our proposed method significantly outperforms the existing representative baselines.

With the development of deep learning in EEG-related tasks, the complexity of learning models has gradually increased. These complex models often result in long inference times, high energy consumption, and an increased risk of overfitting. Therefore, model compression has become an important consideration. Although some EEG models have used lightweight techniques, such as separable convolution, no existing work has directly attempted to compress EEG models to reduce their complexity. In this paper, we integrate neuroscience knowledge into EEG model pruning recovery, and innovatively propose two loss functions in the learning process, the knowledge-guided region-wise loss that enforces the classification evidence consistent with the importance of the prefrontal lobe, and the knowledge-guided sample-wise loss that constrains the learning process by distinguishing the importance of different samples.

3D human pose estimation has been researched for decades with promising fruits. 3D human pose lifting is one of the promising research directions toward the task where both estimated pose and ground truth pose data are used for training. Existing pose lifting works mainly focus on improving the performance of estimated pose, but they usually underperform when testing on the ground truth pose data. We observe that the performance of the estimated pose can be easily improved by preparing good quality 2D pose, such as fine-tuning the 2D pose or using advanced 2D pose detectors. As such, we concentrate on improving the 3D human pose lifting via ground truth data for the future improvement of more quality estimated pose data. Towards this goal, a simple yet effective model called Global-local Adaptive Graph Convolutional Network (GLA-GCN) is proposed in this work. Our GLA-GCN globally models the spatiotemporal structure via a graph representation and backtraces local joint features for 3D human pose estimation via individually connected layers. To validate our model design, we conduct extensive experiments on three benchmark datasets: Human3.6M, HumanEva-I, and MPI-INF-3DHP. Experimental results show that our GLA-GCN1implemented with ground truth 2D poses significantly outperforms state-of-the-art methods (e.g., up to 3%, 17%, and 14% error reductions on Human3.6M, HumanEva-I, and MPI-INF-3DHP, respectively).

Sharing learned models is crucial in research and the industry’s rapid development and progress. Meanwhile, as the Intellectual Property (IP) of the model proposer, the learned high-performance models must be protected to avoid being illegally copied or redistributed by malicious users. Unfortunately, even though the field of Electroencephalography (EEG) has made significant progress and the models are becoming increasingly complex, more work still needs to be done on protecting EEG-based models. The damage caused by model stealing and attack on the brain-computer interface (BCI) is more severe than in other fields. In this paper, we propose a method that protects the IP of EEG-based models with watermarking for the first time. Watermarks are embedded into three representative EEG-based models by designing a trigger set. On the premise of not sacrificing the primary task’s performance significantly, the models’ legality can be verified remotely through the trigger set. Furthermore, we demonstrate that the proposed model protection method is robust to various anti-watermarking attacks, such as fine-tuning, transfer learning, pruning, and watermark overwriting.

With the development of deep learning on EEG-related tasks, the complexity of learning models has gradually increased. Unfortunately, the insufficient amount of EEG data limits the performance of complex models. Thus, model compression becomes an option to be seriously considered. So far, in EEG- related tasks, although some models used lightweight means such as separable convolution in their models, no existing work has directly attempted to compress the EEG model. In this paper, we try to investigate the state-of-the-art network pruning methods on commonly used EEG models for the emotion recognition task. In this work, we make several surprising observations that contradict common beliefs. Training a pruned model from scratch outperforms fine-tuning a pruned model with inherited weights, which means that the pruned structure itself is more important than the inherited weights. We can ignore the entire pruning pipeline and train the network from scratch using the predefined network architecture. We substantially reduce the computational resource overhead of the model while maintaining accuracy. In the best case, we achieve a 62.3% reduction in model size and a 64.3% reduction in computing operations without accuracy loss.

Simultaneous Localization and Mapping (SLAM) has developed as a fundamental method for intelligent robot perception over the past decades. Most of the existing feature-based SLAM systems relied on traditional hand-crafted visual features and a strong static world assumption, which makes these systems vulnerable in complex dynamic environments. In this paper, we propose a robust monocular SLAM system by combining geometry-based methods with two convolutional neural networks. Specifically, a lightweight deep local feature detection network is proposed as the system front-end, which can efficiently generate keypoints and binary descriptors robust against variations in illumination and viewpoint. Besides, we propose a motion segmentation and depth estimation network for simultaneously predicting pixel-wise motion object segmentation and depth map, so that our system can easily discard dynamic features and reconstruct 3D maps without dynamic objects. The comparison against state-of-the-art methods on publicly available datasets shows the effectiveness of our system in highly dynamic environments.

Human action recognition (HAR) in RGB-D videos has been widely investigated since the release of affordable depth sensors. Currently, unimodal approaches (e.g., skeleton-based and RGB video-based) have realized substantial improvements with increasingly larger datasets. However, multimodal methods specifically with model-level fusion have seldom been investigated. In this article, we propose a model-based multimodal network (MMNet) that fuses skeleton and RGB modalities via a model-based approach. The objective of our method is to improve ensemble recognition accuracy by effectively applying mutually complementary information from different data modalities. For the model-based fusion scheme, we use a spatiotemporal graph convolution network for the skeleton modality to learn attention weights that will be transferred to the network of the RGB modality. Extensive experiments are conducted on five benchmark datasets: NTU RGB+D 60, NTU RGB+D 120, PKU-MMD, Northwestern-UCLA Multiview, and Toyota Smarthome. Upon aggregating the results of multiple modalities, our method is found to outperform state-of-the-art approaches on six evaluation protocols of the five datasets; thus, the proposed MMNet can effectively capture mutually complementary features in different RGB-D video modalities and provide more discriminative features for HAR. We also tested our MMNet on an RGB video dataset Kinetics 400 that contains more outdoor actions, which shows consistent results with those of RGB-D video datasets.

Electroencephalography (EEG)-based affective computing has a scarcity problem. As a result, it is difficult to build effective, highly accurate and stable models using machine learning algorithms, especially deep learning models. Data augmentation has recently shown performance improvements in deep learning models with increased accuracy, stability and reduced overfitting. In this paper, we propose a novel data augmentation framework, named the generative adversarial network-based self-supervised data augmentation (GANSER). As the first to combine adversarial training with self-supervised learning for EEG-based emotion recognition, the proposed framework generates high-quality and high-diversity simulated EEG samples. In particular, we utilize adversarial training to learn an EEG generator and force the generated EEG signals to approximate the distribution of real samples, ensuring the quality of the augmented samples. A transformation operation is employed to mask parts of the EEG signals and force the generator to synthesize potential EEG signals based on the unmasked parts to produce a wide variety of samples. A masking possibility during transformation is introduced as prior knowledge to generalize the classifier for the augmented sample space. Finally, numerous experiments demonstrate that our proposed method can improve emotion recognition with an increase in performance and achieve state-of-the-art results.

Despite achieving success in many domains, deep learning models remain mostly black boxes. However, understanding the reasons behind predictions is quite important in assessing trust, which is fundamental in the EEG analysis task. In this work, we propose to use two representative explanation approaches, including LIME and Grad-CAM, to explain the predictions of a simple convolutional neural network on an EEG-based emotional brain-computer interface. Our results demonstrate the interpretability approaches provide the understanding of which features better discriminate the target emotions and provide insights into the neural processes involved in the model learned behaviors.

Given a user’s historical interaction sequence, online novel recommendation suggests the next novel the user may be interested in. Online novel recommendation is important but underexplored. In this paper, we concentrate on recommending online novels to new users of an online novel reading platform, whose first visits to the platform occurred in the last seven days. We have two observations about online novel recommendation for new users. First, repeat novel consumption of new users is a common phenomenon. Second, interactions between users and novels are informative. To accurately predict whether a user will reconsume a novel, it is crucial to characterize each interaction at a fine-grained level. Based on these two observations, we propose a neural network for online novel recommendation, called NovelNet. NovelNet can recommend the next novel from both the user’s consumed novels and new novels simultaneously. Specifically, an interaction encoder is used to obtain accurate interaction representation considering fine-grained attributes of interaction, and a pointer network with a pointwise loss is incorporated into NovelNet to recommend previously-consumed novels. Moreover, an online novel recommendation dataset is built from a well-known online novel reading platform and is released for public use as a benchmark. Experimental results on the dataset demonstrate the effectiveness of NovelNet 1.

Target-oriented opinion words extraction (TOWE) is a subtask of aspect-based sentiment analysis (ABSA). Given a sentence and an aspect term occurring in the sentence, TOWE extracts the corresponding opinion words for the aspect term. TOWE has two types of instance. In the first type, aspect terms are associated with at least one opinion word, while in the second type, aspect terms do not have corresponding opinion words. However, previous researches trained and evaluated their models with only the first type of instance, resulting in a sample selection bias problem. Specifically, TOWE models were trained with only the first type of instance, while these models would be utilized to make inference on the entire space with both the first type of instance and the second type of instance. Thus, the generalization performance will be hurt. Moreover, the performance of these models on the first type of instance cannot reflect their performance on entire space. To validate the sample selection bias problem, four popular TOWE datasets containing only aspect terms associated with at least one opinion word are extended and additionally include aspect terms without corresponding opinion words. Experimental results on these datasets show that training TOWE models on entire space will significantly improve model performance and evaluating TOWE models only on the first type of instance will overestimate model performance.

With the rapid growth of video data, video summarization is a promising approach to shorten a lengthy video into a compact version. Although supervised summarization approaches have achieved state-of-the-art performance, they require frame-level annotated labels. Such an annotation process is time-consuming and tedious. In this article, we propose a novel deep summarization framework named Deep Semantic and Attentive Network for Video Summarization (DSAVS) that can select the most semantically representative summary by minimizing the distance between video representation and text representation without any frame-level labels. Another challenge associated with video summarization tasks mainly originates from the difficulty of considering temporal information over a long time. Long Short-Term Memory (LSTM) performs well for temporal dependencies modeling but does not work well with long video clips. Therefore, we introduce a self-attention mechanism into our summarization framework to capture the long-range temporal dependencies among the frames. Extensive experiments on two popular benchmark datasets, i.e., SumMe and TVSum, show that our proposed framework outperforms other state-of-the-art unsupervised approaches and even most supervised methods.

Video abnormal event detection is a significant research subject in intelligent surveillance, referring to identifying events that do not conform to expected behaviors. It is a challenging task due to the diversity of video content and the various definition of anomaly depending on contexts. How to grasp the nuances of anomalies from diverse video content under associated contexts? In this paper, we propose a context cueing generative adversarial network (CC-GAN) to scop attention to specific regions of interest (ROIs). Different from existing work, the designed framework takes advantage of the context cues to preserve the associations between ROIs and contexts. By intrinsic learning of static spatial layouts and apparent velocities, context cues can help to understand ROIs. Optimized in a crop-inpainting fashion, CC-GAN learns to generate ROIs matching given spatio-temporal context and discriminate the distorted or incongruous ones, corresponding to abnormal scenes. Compared with the state-of-the-art method and other abnormal event detection approaches, the proposed framework demonstrates its effectiveness, reliability, and good generalization ability across various scenes and events.

Collaborative Filtering (CF) is a prevalent technique in recommender systems.Substantial research focuses on learning the embedding of users and items via exploiting past user-item interactions.Recent years have witnessed the boom of Graph Convolutional Networks (GCNs) on CF.Performing graph convolution iteratively, GCN-based models concatenate/average/sum all outputs from different graph convolution layers to generate the embeddings of users and items.Although the previous methods have been proven effective, the pooling operations in the previous methods fail to consider the outputs from different graph convolution layers have different weights and the weights are related to sequential dependencies from precursor nodes.To resolve the aforementioned problems, in this work, we present a new model, Recurrent Neural Graph Collaborative Filtering (RNGCF), which proposes a sequential dependency construction module to adaptively generate the embeddings.Specifically, the module applies a gated recurrent unit (GRU) to learn the sequential dependencies from precursor nodes and an adaptive gated unit (AGU) to adaptively construct the embeddings based on the sequential dependencies.Extensive experiments on three benchmark datasets show that our model outperforms state-ofthe-art models consistently.Our implementation is available in PyTorch 1 .

Sequential recommendation aims to recommend the next item that a user will likely interact with by capturing the useful sequential patterns from users' historical behaviors.Recently, it has become an important and popular component in various e-commerce platforms.As a successful network, Transformer has been widely used to adaptively capture the dynamics of users' historical behaviors for sequential recommendation.In recommender systems, the size of embedding is usually set to be small.Under small embedding, the dot-product in Transformer may have the limitation on calculating the complex relevance between keys and queries.To address the common but neglected issue, in this paper, we present a new model, Deep Self-Attention for Sequential Recommendation (DSASrec), which proposes a chunking deep attention to compute attention weights.The chunking deep attention has two modules: a deep module and a chunking module.The deep module is used to improve the nonlinearity of the attention function.The chunking module is used to calculate attention weights several times like the multihead attention in Transformer.Extensive experiments on three benchmark datasets show that our model can achieve state-ofthe-art results.Our implementation is available in PyTorch 1 .

Summary Today, thanks to the major breakthrough of sequences to sequences model in the field of natural language, most of the dialogue generation tasks are focused on generating more effective responses. However, the responses proposed by the chat‐bot are only a passive answer or assentation, which does not arouse the desire of people to continue communicating. How to transform the chat robot from a passive reply to an active questioner has become an urgent problem. In this paper, a question generalization method with four types of question proposing schemes are designed, implemented, and tested to automate question generation process. The proposed system is controlled by a probability‐triggered multiple conversion mechanism to actively propose different types of questions. We embed our methods in the mainstream dialogue generation model and demonstrate its effectiveness in dialogue response generalization on a standard dataset. In addition, it achieves good performance in subjective conversational assessment.

While current research on multimedia is essentially dealing with the information derived from our observations of the world, internal activities inside human brains, such as imaginations and memories of past events etc., could become a brand new concept of multimedia, for which we coin as “brain-media”. In this paper, we pioneer this idea by directly applying natural images to stimulate human brains and then collect the corresponding electroencephalogram (EEG) sequences to drive a deep framework to learn and visualize the corresponding brain activities. By examining the relevance between the visualized image and the stimulation image, we are able to assess the performance of our proposed deep framework in terms of not only the quality of such visualization but also the feasibility of introducing the new concept of “brain-media”. To ensure that our explorative research is meaningful, we introduce a dually conditioned learning mechanism in the proposed deep framework. One condition is analyzing EEG sequences through deep learning to extract a more compact and class-dependent brain features via exploiting those unique characteristics of human brains such as hemispheric lateralization and biological neurons myelination (neurons importance), and the other is to analyze the content of images via computing approaches and extract representative visual features to exploit artificial intelligence in assisting our automated analysis of brain activities and their visualizations. By combining the brain feature space with the associated visual feature space of those images that are candidates of the stimuli, we are able to generate a combined-conditional space to support the proposed dual-conditioned and lateralization-supported GAN framework. Extensive experiments carried out illustrate that our proposed deep framework significantly outperforms the existing relevant work, indicating that our proposed does provide a good potential for further research upon the introduced concept of “brain-media”, a new member for the big family of multimedia. To encourage more research along this direction, we make our source codes publicly available for downloading at GitHub.

Aspect-category sentiment analysis (ACSA) aims to predict sentiment polarities of sentences with respect to given aspect categories.To detect the sentiment toward a particular aspect category in a sentence, most previous methods first generate an aspect categoryspecific sentence representation for the aspect category, then predict the sentiment polarity based on the representation.These methods ignore the fact that the sentiment of an aspect category mentioned in a sentence is an aggregation of the sentiments of the words indicating the aspect category in the sentence, which leads to suboptimal performance.In this paper, we propose a Multi-Instance Multi-Label Learning Network for Aspect-Category sentiment analysis (AC-MIMLLN), which treats sentences as bags, words as instances, and the words indicating an aspect category as the key instances of the aspect category.Given a sentence and the aspect categories mentioned in the sentence, AC-MIMLLN first predicts the sentiments of the instances, then finds the key instances for the aspect categories, finally obtains the sentiments of the sentence toward the aspect categories by aggregating the key instance sentiments.Experimental results on three public datasets demonstrate the effectiveness of AC-MIMLLN 1 .

Steganographer detection aims to find guilty users who hide secret information in images or other multimedia data in the social network. In existing work, the distances between users are calculated based on the distributions of all images shared by the corresponding users, then users lying an abnormal distance from others are detected as guilty users. This flattened method is difficult to grasp the nuances of the guilty and innocent users. In this paper, we are the first to propose a graph-based deep learning framework for steganographer detection. The proposed Enhancement-aware Graph Convolutional Network (EGCN) represents each user as a weighted complete graph and learns to highlight the differences between guilty users and innocent users based on the structured graph. Compared with the state-of-the-art method and other representative graph-based models, the proposed framework demonstrates its effectiveness across image domains, and even under the context of large-scale social media scenario.

Video summarization technique is a promising approach to process large-scale video by shortening the video content into a compact version. Most previous methods used recurrent networks such as Long Short-Term Memory (LSTM) even combined with attention mechanism to achieve state-of-the-art results. However, these networks are complex to implement and cannot be easily parallelized in computation. In this paper, we propose a novel deep summarization framework named Bi-Directional Self-Attention with Relative Positional Encoding for Video Summarization (BiDAVS) that can be highly parallelized. Our proposed BiDAVS considers position information of input sequence and effectively capture long-range temporal dependencies of sequential frames by computing bi-directional attention. Extensive experiments on two popular benchmark datasets, i.e., SumMe and TVSum, show that our proposed model outperforms state-of-the-art approaches.

Essentially, the current concept of multimedia is limited to presenting what people see in their eyes. What people think inside brains, however, remains a rich source of multimedia, such as imaginations of paradise and memories of good old days etc. In this paper, we propose a dual conditioned and lateralization supported GAN (DCLS-GAN) framework to learn and visualize the brain thoughts evoked by stimulating images and hence enable multimedia to reflect not only what people see but also what people think. To reveal such a new world of multimedia inside human brains, we coin such an attempt as "brain-media". By examining the relevance between the visualized image and the stimulation image, we are able to measure the efficiency of our proposed deep framework regarding the quality of such visualization and also the feasibility of exploring the concept of "brain-media". To ensure that such extracted multimedia elements remain meaningful, we introduce a dually conditioned learning technique in the proposed deep framework, where one condition is analyzing EEGs through deep learning to extract a class-dependent and more compact brain feature space utilizing the distinctive characteristics of hemispheric lateralization and brain stimulation, and the other is to extract expressive visual features assisting our automated analysis of brain activities as well as their visualizations aided by artificial intelligence. To support the proposed GAN framework, we create a combined-conditional space by merging the brain feature space with the visual feature space provoked by the stimuli. Extensive experiments are carried out and the results show that our proposed deep framework significantly outperforms the representative existing state-of-the-arts under several settings, especially in terms of both visualization and classification of brain responses to the evoked images. For the convenience of research dissemination, we make the source code openly accessible for downloading at GitHub.

Anxiety is the most common mental problem that affects nearly 300 million individuals worldwide. The situation is even worse recently. In clinical practice, music therapy has been used for more than forty years because of its effectiveness and few side effects in emotion regulation. This paper proposes a novel style transfer model to generate the therapeutic music according to user's preference. It is widely recognized that the favorite music greatly increases the engagement of the user, hence results in much better curative effects. But in general, users can provide only one or several favorite songs, which are insufficient for the customization of therapeutic music. To address this difficulty, a new domain adaption algorithm that transfers the learning result for music genre classification to the music personalization, is designed. Targeting the joint minimization of the loss functions, three convolutional neural networks are utilized to generate the therapeutic music with only one labelled data of favorite song. The experiment on the anxiety suffers shows that the customized therapeutic music has achieved better and stable performance in anxiety reduction.

As a key component in simultaneous localization and mapping (SLAM) system, loop closure detection (LCD) eliminates the accumulated errors by recognizing previously visited places. In recent years, deep learning methods have been proved effective in LCD. However, most of the existing methods do not make good use of the useful information provided by monocular images, which tends to limit their performance in challenging dynamic scenarios with partial occlusion by moving objects. To this end, we propose a novel workflow, which is able to combine multiple information provided by images. We first introduce semantic information into LCD by developing a local-aware Class Activation Maps (CAMs) weighting method for extracting features, which can reduce the adverse effects of moving objects. Compared with previous methods based on semantic segmentation, our method has the advantage of not requiring additional models or other complex operations. In addition, we propose two effective temporal constraint strategies, which utilize the relationship of image sequences to improve the detection performance. Moreover, we propose to use the keypoint matching strategy as the final detector to further refuse false positives. Experiments on four publicly available datasets indicate that our approach can achieve higher accuracy and better robustness than the state-of-the-art methods.

Developing a dialogue response generation system is one of important topics in natural language processing, but many obstacles are yet to be overcome before autogenerated dialogues with a human-like quality can become possible. A good evaluation method will help narrow the gap between machines and humans in dialogue generation. Unfortunately, the existing automatic evaluation methods are biased and correlate very poorly with human judgments of response quality. Such methods are incapable of assessing whether a dialogue response generation system can produce high-quality, knowledge-related and informative dialogues. In response to this challenge, we design an information-oriented framework to simulate human subjective evaluation. Using this framework, we implement a learning-based metric to evaluate the quality of a dialogue. An experimental validation demonstrates our proposed metric's effectiveness in dialogue selection and model evaluation on a Twitter dataset (in English) and a Weibo dataset (in Chinese). In addition, the metric is more relevant than the existing methods of dialogue evaluation to human subjective judgment.

Image steganalysis is to discriminate innocent images (cover images) and those suspected images (stego images) with hidden messages. The task is challenging since modifications to cover images due to message hiding are extremely small. To handle this difficulty, modern approaches proposed using convolutional neural network (CNN) models to detect steganography with paired learning, i.e., cover images and their stegos are both in training set. In this paper, we explore an important technique in CNN models, the batch normalization (BN), for the task of image steganalysis in the paired learning framework. Our theoretical analysis shows that a CNN model with multiple batch normalization layers is difficult to be generalized to new data in the test set when it is well trained with paired learning. To address this problem, we propose a novel normalization technique called shared normalization (SN) in this paper. Unlike the BN layer utilizing the mini-batch mean and standard deviation to normalize each input batch, SN shares consistent statistics for training samples. Based on the proposed SN layer, we further propose a novel neural network model for image steganalysis. Extensive experiments demonstrate that the proposed network with SN layers is stable and can detect the state-of-the-art steganography with better performances than previous methods.

Steganographer detection aims to identify the guilty user who utilizes steganographic methods to hide secret information in the spread of multimedia data, especially image data, from a large amount of innocent users on social networks. A true embedding probability map illustrates the probability distribution of embedding secret information in the corresponding images by specific steganographic methods and settings, which has been successfully used as the guidance for content-adaptive steganographic and steganalytic methods. Unfortunately, in real-world situation, the detailed steganographic settings adopted by the guilty user cannot be known in advance. It thus becomes necessary to propose an automatic embedding probability estimation method. In this article, we propose a novel content-adaptive steganographer detection method via embedding probability estimation. The embedding probability estimation is first formulated as a learning-based saliency detection problem and the multi-scale estimated map is then integrated into the CNN to extract steganalytic features. Finally, the guilty user is detected via an efficient Gaussian vote method with the extracted steganalytic features. The experimental results prove that the proposed method is superior to the state-of-the-art methods in both spatial and frequency domains.

Multi-video summarization, which tries to generate a single summary for a collection of video, is an important task in dealing with ever-growing video data. In this paper, we are the first to propose a graph convolutional network for multi-video summarization. The novel network measures the importance and relevance of each video shot in its own video as well as in the whole video collection. The important node sampling method is proposed to emphasize the effective features which are more possible to be selected as the final video summary. Two strategies are proposed to integrate into the network to solve the inherent class imbalance problem in the task of video summarization. The loss regularization for diversity is used to encourage a diverse summary to be generated. Extensive experiments are carried out, and in comparison with traditional and recent graph models and the state-of-the-art video summarization methods, our proposed model is effective in generating a representative summary for multiple videos with good diversity. It also achieves state-of-the-art performance on two standard video summarization datasets.

Multimedia stimulation of brain activities is not only becoming an emerging area for intensive research, but also achieved significant progresses towards classification of brain activities and interpretation of brain understanding of multimedia content. To exploit the characteristics of EEG signals in capturing human brain activities, we propose a region-dependent and attention-driven bi-directional LSTM network (RA-BiLSTM) for image evoked brain activity classification. Inspired by the hemispheric lateralization of human brains, the proposed RA-BiLSTM extracts additional information at regional level to strengthen and emphasize the differences between two hemispheres. In addition, we propose a new attentional-LSTM by adding an extra attention gate to: (i) measure and seize the importance of channel-based spatial information, and (ii) support the proposed RA-BiLSTM to capture the dynamic correlations hidden from both the past and the future in the current state across EEG sequences. Extensive experiments are carried out and the results demonstrate that our proposed RA-BiLSTM not only achieves effective classification of brain activities on evoked image categories, but also significantly outperforms the existing state of the arts.

Over the past decade, “content-based” multimedia systems have realized success. By comparison, brain imaging and classification systems demand more efforts for improvement with respect to accuracy, generalization, and interpretation. The relationship between electroencephalogram (EEG) signals and corresponding multimedia content needs to be further explored. In this paper, we integrate implicit and explicit learning modalities into a context-supported deep learning framework. We propose an improved solution for the task of brain imaging classification via EEG signals. In our proposed framework, we introduce a consistency test by exploiting the context of brain images and establishing a mapping between visual-level features and cognitive-level features inferred based on EEG signals. In this way, a multimodal approach can be developed to deliver an improved solution for brain imaging and its classification based on explicit learning modalities and research from the image processing community. In addition, a number of fusion techniques are investigated in this work to optimize individual classification results. Extensive experiments have been carried out, and their results demonstrate the effectiveness of our proposed framework. In comparison with the existing state-of-the-art approaches, our proposed framework achieves superior performance in terms of not only the standard visual object classification criteria, but also the exploitation of transfer learning. For the convenience of research dissemination, we make the source code publicly available for downloading at GitHub (https://github.com/aneeg/dual-modal-learning).

Dialogue response generation system is one of the hot topics in natural language processing, but it is still a long way to go before it can generate human-like dialogues. A good evaluation method will help narrow the gap between the machine and human in dialogue generation. Unfortunately, current evaluation methods cannot measure whether the dialogue response generation system is able to produce high-quality, knowledge-related, and informative dialogues. Aiming to identify and measure the existence of information in dialogues, we propose a novel automatic evaluation metric. By learning from the knowledge representation method in knowledge base, we define the heuristic rules to extract the information triples from dialogue pairs. And we design an information matching method to measure the probability of the existence of information in a dialogue. In experiments, our proposed metric demonstrates its effectiveness in dialogue selection and model evaluation on the Reddit dataset (English) and the Weibo dataset (Chinese).

Steganographer detection task is to identify criminal users, who attempt to conceal confidential information by steganography methods, among a large number of innocent users. The significant challenge of the task is how to collect the evidences to identify the guilty user with suspicious images, which are embedded with secret messages generating by unknown steganography and payload. Unfortunately, existing methods for steganalysis were served for the binary classification. It makes them harder to classify the images with different kinds of payloads, especially when the payloads of images in test dataset have not been provided in advance. In this paper, we propose a novel steganographer detection method based on multiclass deep neural networks. In the training stage, the networks are trained to classify the images with six types of payloads. The networks can preserve even strengthen the weak stego signals from secret messages in much larger receptive filed by virtue of residual and dilated residual learning. In the inference stage, the learnt model is used to extract the discriminative features, which can capture the difference between guilty users and innocent users. A series of empirical experimental results demonstrate that the proposed method achieves good performance in spatial and frequency domains even though the embedding payload is low. The proposed method achieves a higher level of robustness of inter-steganographic algorithms and can provide a possible solution to address the payload mismatch problem

Algorithmic composition, which enables computer to generate music like human composers, has lasting charm because it intends to approximate artistic creation, most mysterious part of human intelligence. To deliver both melodious and refreshing music, this paper proposes the Musicality-Novelty Generative Adversarial Nets for algorithmic composition. With the same generator, two adversarial nets alternately optimize the musicality and novelty of the machine-composed music. A new model called novelty game is presented to maximize the minimal distance between the machine-composed music sample and any human-composed music sample in the novelty space, where all well-known human composed music products are far from each other. We implement the proposed framework using three supervised CNNs with one for generator, one for musicality critic and one for novelty critic on the time-pitch feature space. Specifically, the novelty critic is implemented by Siamese neural networks with temporal alignment using dynamic time warping. We provide empirical validations by generating the music samples under various scenarios.

Image steganalysis is to discriminate innocent images and those suspected images with hidden messages. In this paper, we propose a unified Convolutional Neural Network (CNN) model for this task. In order to reliably detect modern steganographic algorithms, we design the proposed model from two aspects. For the first, different from existing CNN based steganalytic algorithms that use a predefined highpass kernel to suppress image content, we integrate the highpass filtering operation into the proposed network by building a content suppression subnetwork. For the second, we propose a novel sub-network to actively preserve the weak stego signal generated by secret messages based on residual learning, making the successive network capture the difference between cover images and stego images. Extensive experiments demonstrate that the proposed model can detect states-of-the-art steganography with much lower detection error rates than previous methods.

Steganographer detection problem is to identify culprit actors, who try to hide confidential information with steganography, among many innocent actors. This task has significant challenges, including various embedding steganographic algorithms and payloads, which are usually avoided in steganalysis under laboratory conditions. In this paper, we propose a novel steganographer detection model based on deep residual network. The proposed method strengthens the signal coming from secret messages, which is beneficial for the discrimination between guilty actors and innocent actors. Comprehensive experiments demonstrate that the proposed model achieves very low detection error rates in steganographer detection task. It also outperforms the classical rich model method and other CNN based method. Moreover, the model shows the robustness of inter-steganographic algorithms and inter-payloads.

According to consciousness involvement, human’s learning can be roughly classified into explicit learning and implicit learning. Contrasting strongly to explicit learning with clear targets and rules, such as our school study of mathematics, learning is implicit when we acquire new information without intending to do so. Research from psychology indicates that implicit learning is ubiquitous in our daily life. Moreover, implicit learning plays an important role in human visual perception. But in the past 60 years, most of the well-known machine-learning models aimed to simulate explicit learning while the work of modeling implicit learning was relatively limited, especially for computer vision applications. This article proposes a novel unsupervised computational model for implicit visual learning by exploring dissipative system, which provides a unifying macroscopic theory to connect biology with physics. We test the proposed Dissipative Implicit Learning Model (DILM) on various datasets. The experiments show that DILM not only provides a good match to human behavior but also improves the explicit machine-learning performance obviously on image classification tasks.

Recent studies have demonstrated that a well designed deep convolutional neural network (CNN) model achieves competitive performances on detecting the presence of secret message in digital images, compared with the classical rich model based steganalysis. In this paper, we propose to investigate a category of very deep CNN model-the deep residual network (DRN), for steganalysis. DRN is suitable for steganalysis from two aspects. For the first, the DRN model usually contains a large number of network layers, which proves to be effective to capture the complex statistics of digital images. For the second, DRN's residual learning (ResL) method actively strengthens the signal coming from secret messages, which is extremely beneficial for the discrimination between cover images and stego images. Comprehensive experiments on standard dataset show that the DRN model achieves very low detection error rates for the state of arts steganographic algorithms. It also outperforms the classical rich model method and several recently proposed CNN based methods.

The details of oriented visual stimuli are better resolved when they are horizontal or vertical rather than oblique. This "oblique effect" has been researched and confirmed in numerous research studies, including behavioral studies and neurophysiological and neuroimaging findings. Although the "oblique effect" has influence in many fields, little research integrated it into computational models. In this paper, we try to explore this inhomogeneity of visual orientation based on Convolutional neural networks (CNNs) in image recognition. We validate that visual orientation inhomogeneity CNNs can achieve comparable performance with higher computational efficiency on various datasets. We can also get the conclusion that, compared with the cardinal information, oblique information is indeed less useful in natural color image recognition. Through the exploration of the proposed model on image recognition, we gain more understanding of the inhomogeneity of visual orientation. It also illuminates a wide range of opportunities for integrating the inhomogeneity of visual orientation with other computational models.

Image recognition with incomplete data is a well-known hard problem in computer vision and machine learning. This article proposes a novel deep learning technique called Field Effect Bilinear Deep Networks (FEBDN) for this problem. To address the difficulties of recognizing incomplete data, we design a novel second-order deep architecture with the Field Effect Restricted Boltzmann Machine, which models the reliability of the delivered information according to the availability of the features. Based on this new architecture, we propose a new three-stage learning procedure with field effect bilinear initialization, field effect abstraction and estimation, and global fine-tuning with missing features adjustment. By integrating the reliability of features into the new learning procedure, the proposed FEBDN can jointly determine the classification boundary and estimate the missing features. FEBDN has demonstrated impressive performance on recognition and estimation tasks in various standard datasets.

The large-scale video data on the web contain a lot of semantics, which are an important part of semantic web. Video descriptors can usually represent somewhat the semantics. Thus, they play a very important role in web multimedia content analysis, such as Scale-invariant feature transform (SIFT) feature. In this paper, we proposed a new video descriptor, called a temporal-compress and shorter SIFT(TC-S-SIFT) which can efficiently and effectively represent the semantics of web videos. By omitting the least discriminability orientation in three stages of standard SIFT on every representative frame, the dimensions of the shorter SIFT are reduced from 128-dimension to 96-dimension to save space storage. Then, the SIFT can be compressed by tracing SIFT features on video temporal domain, which highly compress the quantity of local features to reduce visual redundancy, and keep basically the robustness and discrimination. Experimental results show our method can yield comparable accuracy and compact storage size.

Human vision system actively seeks salient regions and movements in video sequences to reduce the search effort. Modeling computational visual saliency map provides im-portant information for semantic understanding in many real world applications. In this paper, we propose a novel video saliency detection model for detecting the attended regions that correspond to both interesting objects and dominant motions in video sequences. In spatial saliency map, we in-herit the classical bottom-up spatial saliency map. In tem-poral saliency map, a novel optical flow model is proposed based on the dynamic consistency of motion. The spatial and the temporal saliency maps are constructed and further fused together to create a novel attention model. The pro-posed attention model is evaluated on three video datasets. Empirical validations demonstrate the salient regions de-tected by our dynamic consistent saliency map highlight the interesting objects effectively and efficiency. More im-portantly, the automatically video attended regions detected by proposed attention model are consistent with the ground truth saliency maps of eye movement data.

Water reflection, a typical imperfect reflection symmetry problem, plays an important role in image content analysis. Existing techniques of symmetry recognition, however, cannot recognize water reflection images correctly because of the complex and various distortions caused by the water wave. Hence, we propose a novel water reflection recognition technique to solve the problem. First, we construct a novel feature space composed of motion blur invariant moments in low-frequency curvelet space and of curvelet coefficients in high-frequency curvelet space. Second, we propose an efficient algorithm including two sub-algorithms: low-frequency reflection cost minimization and high-frequency curvelet coefficients discrimination to classify water reflection images and to determine the reflection axis. Through experimenting on authentic images in a series of tasks, the proposed techniques prove effective and reliable in classifying water reflection images and detecting the reflection axis, as well as in retrieving images with water reflection.

Extractive style query-oriented multi-document summarization generates the summary by extracting a proper set of sentences from multiple documents based on the pre-given query. This paper proposes a novel multi-document summarization framework via deep learning model. This uniform framework consists of three parts: concepts extraction, summary generation, and reconstruction validation, which work together to achieve the largest coverage of the documents content. A new query-oriented extraction technique is proposed to concentrate distributed information to hidden units layer by layer. Then, the whole deep architecture is fine-tuned by minimizing the information loss of reconstruction validation. According to the concentrated information, dynamic programming is used to seek most informative set of sentences as the summary. Experiments on three benchmark datasets demonstrate the effectiveness of the proposed framework and algorithms.

Image recognition with incomplete data is a well-known hard problem in multimedia content analysis. This paper proposes a novel deep learning technique called semiconducting bilinear deep belief networks (SBDBN) by referencing human's visual cortex and intelligent perception. Inheriting from deep models, SBDBN simulates the laminar structure of human's cerebral cortex and the neural loop in human's visual areas. To address the special difficulties of image recognition with incomplete data, we design a novel second-order deep architecture with semiconducting restricted boltzmann machines. Moreover, two peaks activation of human's perception is implemented by three learning stages of semiconducting bilinear discriminant initialization, greedy layer-wise reconstruction, and global fine-tuning. Owing to exploiting the embedding information according to the reliable features rather than any completion of missing features, the proposed SBDBN has demonstrated outstanding recognition ability on two standard datasets and one constructed dataset, comparing with both incomplete image recognition techniques and existing deep learning models.

Detection and description of local features are a classical problem in image processing and multimedia content analysis. Based on the in homogeneity of visual orientation in human visual system, we propose a novel algorithm S-SIFT to detect and describe local image features. In three stages of S-SIFT, the information from the least discriminability orientation is omitting. Compared with the standard SIFT algorithm, S-SIFT has lower dimension and provides a faster key point matching. Experiments on the standard dataset demonstrate that our algorithm yields comparable or even better results for feature detection and matching tasks.

Water reflection, a kind of typical imperfect reflection symmetry problem, plays an important role in image content analysis. However, existing techniques of symmetry recognition cannot recognize water reflection images correctly because of the complex and various distortions caused by water wave. To address this difficulty, we construct a novel feature space which is composed of motion blur invariant moments. Moreover, we propose an efficient detection algorithm to determine the reflection axis in images with water reflection. By experimenting on real image dataset with different tasks, the proposed techniques demonstrate impressive results in the water reflection image classification, the reflection axis detection, and the retrieval of the images with water reflection.

In this paper, we present a novel algorithm called manifold ordinal regression (MOR) for image ranking. By modeling the manifold information in the objective function, MOR is capable of uncovering the intrinsically nonlinear structure held by the image data sets. By optimizing the ranking information of the training data sets, the proposed algorithm provides faithful rating to the new coming images. To offer more general solution for the real-word tasks, we further provide the semi-supervised manifold ordinal regression (SS-MOR). Experiments on various data sets validate the effectiveness of the proposed algorithms.

Image classification is a well-known classical problem in multimedia content analysis. This paper proposes a novel deep learning model called bilinear deep belief network (BDBN) for image classification. Unlike previous image classification models, BDBN aims to provide human-like judgment by referencing the architecture of the human visual system and the procedure of intelligent perception. Therefore, the multi-layer structure of the cortex and the propagation of information in the visual areas of the brain are realized faithfully. Unlike most existing deep models, BDBN utilizes a bilinear discriminant strategy to simulate the "initial guess" in human object recognition, and at the same time to avoid falling into a bad local optimum. To preserve the natural tensor structure of the image data, a novel deep architecture with greedy layer-wise reconstruction and global fine-tuning is proposed. To adapt real-world image classification tasks, we develop BDBN under a semi-supervised learning framework, which makes the deep model work well when labeled images are insufficient. Comparative experiments on three standard datasets show that the proposed algorithm outperforms both representative classification models and existing deep learning techniques. More interestingly, our demonstrations show that the proposed BDBN works consistently with the visual perception of humans.

No abstract available.

This work presents a semantic level no-reference image sharpness/blurriness metric under the guidance of top-down & bottom-up saliency map, which is learned based on eye-tracking data by SVM. Unlike existing metrics focused on measuring the blurriness in vision level, our metric more concerns about the image content and human's intention. We integrate visual features, center priority, and semantic meaning from tag information to learn a top-down & bottom-up saliency model based on the eye-tracking data. Empirical validations on standard dataset demonstrate the effectiveness of the proposed model and metric.