Multi-modal object Re-identification refers to the task of identifying the same object from different cameras according to multi-modal cues. The existing multi-modal object Re-ID methods have achieved remarkable progress in extracting discriminative global and local representations. However, confined to learning generic static features, these approaches lack the relational structures and deep semantic reasoning needed for complex alignment, rendering them susceptible to modality noise as they passively aggregate information without filtering interference. To solve these problems, an LSR2 framework is imposed to actively mine relational and reasoning patterns. Specifically, the Representation Learning with Frequency Mining (RFM) module purifies features via frequency-domain decoupling. The Relational Expert System with Selective Reasoning (RER) then leverages expert guidance to adaptively filter relational cues. Finally, a Structural Consistency Constraint (SCC) enforces topological alignment to reduce high-dimensional mapping ambiguity. Extensive experiments on three object re-identification benchmark sets verify that by successfully learning rich relational and reasoning representations, the proposed method overcomes traditional modality noise and achieves superior performance.

Comparative Opinion Quintuple Extraction (COQE) aims to extract all comparative sentiment quintuples from product review text. Each quintuple comprises five elements: subject, object, aspect, opinion and preference. With the rise of Large Language Models (LLMs), existing work primarily focuses on enhancing the performance of COQE task through data augmentation, supervised fine-tuning and instruction tuning. Instead of the above pre-modeling and in-modeling design techniques, we focus on innovation in the post-processing. We introduce a model-unaware adaptive chain-of-feedback (COF) method from the perspective of inference feedback and extraction revision. This method comprises three core modules: dynamic example selection, self-critique and self-revision. By integrating LLMs, COF enables dynamic iterative self-optimization, making it applicable across different baselines. To validate the effectiveness of our approach, we utilize the outputs of two distinct baselines as inputs for COF: frozen parameters few-shot learning and the SOTA supervised fine-tuned model. We evaluate our approach on three benchmarks: Camera, Car and Ele. Experimental results show that, compared to the few-shot learning method, our approach achieves F1 score improvements of 3.51%, 2.65% and 5.28% for exact matching on the respective dataset. Even more impressively, our method further boosts performance, surpassing the current SOTA results, with additional gains of 0.76%, 6.54%, and 2.36% across the three datasets.

Large language model progress drives code generation research. Most works enhance performance via problem decomposition and knowledge distillation. However, they predominantly adopt a unidirectional approach to knowledge learning. Inspired by the mutually beneficial learning patterns observed in humans, this paper explores a mutual learning paradigm. Specifically, we first employ a dual-training architecture involving two lightweight models that learn the knowledge from each other for better performance. Then, we maximize mutual learning potential by exploring mutual self-correction and preference optimization to address knowledge sharing limits. The self-correction strategy incorporates error information generated by both the model itself and its peer, fostering a mutual refinement process. The preference optimization utilizes the excellent knowledge that has not been learned from the peer model to guide the learning. We conduct and analyze the proposed method on two public datasets, expanding the corpus for optimization. Experimental results show the effectiveness of our method, yielding compelling outcomes that underscore the potential of this collaborative learning approach.

Cross-domain Aspect Sentiment Triplet Extraction (ASTE) aims to extract the triplets from the review of a target domain, utilizing knowledge from a source domain. As a newly proposed task, limited work has been devoted to it. Except for solving it in a zero-shot manner with in-domain models, recent work explores a bidirectional generative framework to generate pseudo-labeled target data. However, such a method suffers from low efficiency with two-stage training and unstable pseudo-label quality. In this paper, we propose a Hybrid Prompts Mixture (HiPM) method for cross-domain ASTE to fully utilize domain-independent knowledge. Within this method, given that syntax information is an essential linguistic feature for triplet extraction, we design a syntax-related hard prompt to transfer the structures. Additionally, aspects from different domains exhibit similarities in their respective categories. We take this shared information as the prototypes and enrich them through a warm-up step. The resulting prototypes then act as the source of soft prompts. We further mix the hard and soft prompts with the original sequence into a generative model to extract triplets. Experimental results show that our method outperforms baselines on twelve transfer pairs, and obtains a 1.48% average F1 score improvement over the state-of-the-art cross-domain ASTE model.

In the field of Natural Language Processing (NLP), Aspect-Based Sentiment Analysis (ABSA) has gained significant attention in recent years due to its ability to perform fine-grained sentiment analysis. Generative methods tackle various ABSA tasks by autoregressively generating the target sequence of sentiment tuples in a specified format. However, the sentiment tuple is intrinsically an unordered set, and the method introduces an order bias between the generated sequence and the original target. Therefore, to investigate the impact of sentiment tuples order on model performance, we conduct a pilot experiment, unveiling that the order of tuples significantly influences the learning outcomes of the Seq2Seq model. Thus, we propose a novel tuple-order learning method that prioritizes tuples from simple to complex, facilitated by a discrete evaluation method that assesses the difficulty of each individual tuple. Specifically, we incorporate positional information on tuples and employ an effective strategy to expedite the assessment of individual tuples. The method optimizes the learning process while maintaining the structural integrity of existing generative models. Extensive experiments show that our approach significantly advances the performance on 14 datasets of 5 benchmark tasks. We will release our code at https://github.com/gongzhenhu/TOL.

Multi-party dialogue reading comprehension is an extraction-based reading comprehension task that aims to understand dialogue with multiple interlocutors and answer related questions. The frequent rotation of topics and the irregular order of interlocutors in dialogues may lead to the scattered distribution of information in multi-party dialogues. This means that the model needs to effectively integrate information across multiple utterances and among various interlocutors. Although previous methods have made considerable efforts in mining and modeling dialogue-related features, they still encounter two key issues. On the one hand, these methods failed to solve the cross-utterance co-reference problem that arises from the coexistence of multiple topics and interlocutors in the dialogue. On the other hand, they mostly ignored the joint reasoning of multi-granularity dialogue-related features, which can parse the semantic space of multi-party dialogue from coarse to fine. To overcome these bottlenecks, we propose a dual-granularity information joint reasoning method, which performs hierarchically semantic modeling for multi-party dialogue based on the graph attention networks. Specifically, we utilize discourse dependency relationships and interlocutor-aware temporal information to conduct coarsegrained semantic modeling, and perform fine-grained semantic refinement by leveraging token-level co-reference relationships. Our method demonstrates stable and substantial performance improvement when using different pre-trained language models as backbones and achieves a new state-of-the-art on the benchmark corpora Molweni and FriendsQA.

Causal reasoning is vitally important for various natural language processing, which needs text semantic understanding and rich knowledge information reserve. Causal question-answering (CQA), one of the causal reasoning tasks, aims to choose either the cause or effect of a given story sentence. It requires both background causal knowledge and the ability to infer cause-effect relations. However, existing studies ignore the logical and commonsense relationship between the contexts, which limits the model capability. In this paper, we propose a novel model of Semantic Internal-External Enhancement (SIEE) by enhancing both the internal and external knowledge. The model employs Abstract Meaning Representation (AMR) to capture the core semantic information and explicit structures. In addition, we explore the commonsense knowledge behind the key information in the context to provide more clues for reasoning. Finally, we combine the above internal and external information by using a semantic aggregator to aggregate the semantic information of neighbors on the keyword nodes. Experimental studies show the competitive performance of our proposed model over the state-of-the-art published results on three CQA benchmarks, e-CARE, COPA and BCOPA.

Current self-training methods such as standard self-training, co-training, tri-training, and others often focus on improving model performance on a single task, utilizing differences in input features, model architectures, and training processes.However, many tasks in natural language processing are about different but related aspects of language, and models trained for one task can be great teachers for other related tasks.In this work, we propose friendtraining, a cross-task self-training framework, where models trained to do different tasks are used in an iterative training, pseudo-labeling, and retraining process to help each other for better selection of pseudo-labels.With two dialogue understanding tasks, conversational semantic role labeling and dialogue rewriting, chosen for a case study, we show that the models trained with the friend-training framework achieve the best performance compared to strong baselines.

Existing approaches to Emotion Recognition in Conversation (ERC) use a fixed context window to recognize speakers’ emotion, which may lead to either scantiness of key context or interference of redundant context. In response, we explore the benefits of variable-length context and propose a more effective approach to ERC. In our approach, we leverage different context windows when predicting the emotion of different utterances. New modules are included to realize variable-length context: 1) two speaker-aware units, which explicitly model inner- and inter-speaker dependencies to form distilled conversational context and 2) a top-k normalization layer, which determines the most proper context windows from the conversational context to predict emotion. Experiments and ablation study show that our approach outperforms several strong baselines on three public datasets.

Legal judgment prediction (LJP) is used to predict judgment results based on the description of individual legal cases. In order to be more suitable for actual application scenarios in which the case has cited multiple articles and has multiple charges, we formulate legal judgment prediction as a multiple label learning problem and present a deep learning model that can effectively encode the content of each legal case via a multi-residual convolution neural network and the semantics of law articles via an article encoder. An article-wise attention mechanism is proposed to fuse the two types of encoded information. Experimental results derived on the CAIL2018 datasets show that our model provides a significant performance improvement over the existing neural models in predicting relevant law articles and charges.

Emotion recognition in conversation is one of the essential tasks of natural language processing. However, this task’s annotation data is insufficient since such data is hard to collect and annotate. Meanwhile, there is large-scale data for conversational generation, and this data does not need annotation manually. But, whether the vector space between different datasets is similar will be a problem. Therefore, we utilize a same dataset to train the conversational generator and the classifier, and transfer knowledge between them. In particular, we propose an Emotion Recognition with Conversational Generation Transfer (ERCGT) framework to model the interaction among utterances by transfer learning. First, we train a conversational generator. In the second step, a transfer learning model is used to transfer the knowledge of generator to the emotion recognition model. Empirical studies illustrate the effectiveness of the proposed framework over several strong baselines on three benchmark emotion classification datasets.

In recent years, distantly-supervised relation extraction has achieved a certain success by using deep neural networks.Distant Supervision (DS) can automatically generate large-scale annotated data by aligning entity pairs from Knowledge Bases (KB) to sentences.However, these DSgenerated datasets inevitably have wrong labels that result in incorrect evaluation scores during testing, which may mislead the researchers.To solve this problem, we build a new dataset NYT-H, where we use the DS-generated data as training data and hire annotators to label test data.Compared with the previous datasets, NYT-H has a much larger test set and then we can perform more accurate and consistent evaluation.Finally, we present the experimental results of several widely used systems on NYT-H.The experimental results show that the ranking lists of the comparison systems on the DS-labelled test data and human-annotated test data are different.This indicates that our human-annotated data is necessary for evaluation of distantly-supervised relation extraction.

A current research trend is to combine multimedia data with artificial intelligence and process them on cloud servers. In this context, ensuring the security of multimedia cloud servers is critical, and the cyber mimic defence (CMD) technology is a promising approach to this end. CMD, which is an innovative active defence technology developed in China, can be applied in many scenarios. However, although the mathematical model is a key component of CMD, a universally acceptable mathematical model for theoretical CMD has not been established yet. In this work, the attack problems and modelling difficulties were extensively examined, and a comprehensive modelling theory and concepts were clarified. By decoupling the model from the input and output of the specific system scene, the modelling difficulties were effectively avoided, and the mathematical expression of the CMD mechanism was enhanced. Furthermore, the process characteristics of the attack behaviour were identified by using a specific mathematical mapping method. Finally, based on the decomposition problem of large prime factors and convolution operations, an intuitive and exclusive CMD mathematical model was proposed. The proposed model could clearly express the CMD mechanism and transform the problems of attack and defence in the CMD domain into corresponding mathematical problems. These aspects were considered to qualitatively assess the CMD security, and it was noted that a high level of security can be realized. Furthermore, the overhead of CMD was analyzed. Moreover, the proposed model can be directly programmed.

With the rapid development of the Internet, the security of network multimedia data has attracted increasingly more attention. The moving target defense (MTD) and cyber mimic defense (CMD) approaches provide a new way to solve this problem. To enhance the security of network multimedia data, this paper proposes a mimic encryption box for network multimedia data security. The mimic encryption box can directly access the network where the multimedia device is located, automatically complete the negotiation, provide safe and convenient encryption services, and effectively prevent network attacks. According to the principles of dynamization, diversification, and randomization, the mimic encryption box uses a reconfigurable encryption algorithm to encrypt network data and uses IP address hopping, port number hopping, protocol camouflage, and network channel change to increase the attack threshold. Second, the mimic encryption box has a built-in pseudorandom number generator and key management system, which can generate an initial random key and update the key with the hash value of the data packet to achieve “one packet, one key.” Finally, through the cooperation of the ARM and the FPGA, an access control list can be used to filter illegal data and monitor the working status of the system in real time. If an abnormality is found, the feedback reconstruction mechanism is used to “clean” the FPGA to make it work normally again. The experimental results and analysis show that the mimic encryption box designed in this paper has high network encryption performance and can effectively prevent data leakage. At the same time, it provides a mimic security defense mechanism at multiple levels, which can effectively resist a variety of network attacks and has high security.

Xiabing Zhou, Zhongqing Wang, Shoushan Li, Guodong Zhou, Min Zhang. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Learning with incomplete data remains challenging in many real-world applications especially when the data is high-dimensional and dynamic. Many imputation-based algorithms have been proposed to handle with incomplete data, where these algorithms use statistics of the historical information to remedy the missing parts. However, these methods merely use the structural information existing in the data, which are very helpful for sharing between the complete entries and the missing ones. For example, in traffic system, some group information and temporal smoothness exist in the data structure. In this paper, we propose to incorporate these structural information and develop structural feature leaning method for learning with incomplete data (SFLIC). The SFLIC model adopt a fused Lasso based regularizer and a group Lasso style regularizer to enlarge the data sharing along both the temporal smoothness level and the feature group level to fill the gap where the data entries are missing. The proposed SFLIC model is a nonsmooth function according to the model parameters, and we adopt the smoothing proximal gradient (SPG) method to seek for an efficient solution. We evaluate our model on both synthetic and real-world highway traffic datasets. Experimental results show that our method outperforms the state-of-the-art methods.

Nearest neighbor based nonparametric regression is a classic data-driven method for traffic flow prediction in intelligent transportation systems (ITS). Performances of those models depend heavily on the similarity or distance metric used to search nearest neighborhood. Metric learning algorithms have been developed to learn the distance metrics from data in recent years. In real-world transportation application, multiple forecasting tasks are set since there are lots of road sections and detector points in the traffic network. Previous works tend to learn only one global metric to be used for all the tasks or learn multiple local metrics for each task which may lead to under-fitting or over-fitting problem. To balance these two kinds of methods and improve the generalization of learned metrics, we propose a common metric learning algorithm under the intuition that homogenous tasks tend to have similar local metrics. Then the learned common metrics are used in common metric KNN (CM-KNN) for traffic flow prediction. Experimental results show that our algorithm to learn common metrics are reasonable and CM-KNN method for traffic flow prediction outperforms other competing methods.

Ensemble learning of neural network is a learning paradigm where ensembles of several neural networks show improved generalization capabilities that outperform those of single networks. For deep learning of multi-layer neural networks, ensemble learning is still applicable. In addition, characteristics of deep neural networks can provide potential opportunities to improve the performance of traditional neural network ensembles. In this paper, we propose an ensemble criterion of deep neural networks that is based on the reconstruction error and present two strategies to solve the most important issues in ensemble learning of neural networks: component dataset sampling and output averaging. Component training datasets are selected according to the reconstruction error instead of random bootstrap sampling or re-weighting. Moreover, for each testing instance, we can compute the reconstruction error yielded by the sub-model simultaneously with the output. The reconstruction error is used as the weights in output averaging. From the perspectives of prediction interval and confidence interval, we demonstrated that smaller reconstruction error could ensure smaller prediction interval. We also incorporate the famous structure ensemble approach “Dropout” into the proposed approach to achieve the best performance. We conduct experiments on classification and regression datasets to validate the effectiveness of our approach.

Learning temporal causal structures between time series is one of key tools for analyzing time series data. Most previous works focuse on learning with static temporal causal relationships. However, in many real world applications, such as climate environment and transportation system, the causal structures vary dramatically over time. In this paper, we propose a probabilistic dynamic causal (PDC) model based on Lasso-Granger to uncover the dynamic temporal dependencies. Specifically, the PDC model infers different state varying of temporal data and causal structures of each state in one unified model. We devise the expectation-maximization (EM) algorithm to infer the model parameters. Furthermore, to address the smoothness of state varying in adjacent time, we extend the PDC model with a regularization term encouraging states to be similar in adjacent time. Though it may slightly decrease the precision on training data, it improves the generalization capability of the model. We conduct experiments on synthetic dataset as well as two real-world datasets of climate and traffic to evaluate the effectiveness of the PDC model. Experimental results show that the proposed model is effective in discovering the dynamic causal factors of Particulate Matter 2.5 (PM2.5) and traffic spatial causalities.

With the proliferation of mobile devices and wireless technologies, mobile social network systems are increasingly available. A mobile social network plays an essential role as the spread of information and influence in the form of “word-of-mouth”. It is a fundamental issue to find a subset of influential individuals in a mobile social network such that targeting them initially (e.g., to adopt a new product) will maximize the spread of the influence (further adoptions of the new product). The problem of finding the most influential nodes is unfortunately NP-hard. It has been shown that a Greedy algorithm with provable approximation guarantees can give good approximation; However, it is computationally expensive, if not prohibitive, to run the greedy algorithm on a large mobile social network. In this paper, a divide-and-conquer strategy with parallel computing mechanism has been adopted. We first propose an algorithm called Community-based Greedy algorithm for mining top-K influential nodes. It encompasses two components: dividing the large-scale mobile social network into several communities by taking into account information diffusion and selecting communities to find influential nodes by a dynamic programming. Then, to further improve the performance, we parallelize the influence propagation based on communities and consider the influence propagation crossing communities. Also, we give precision analysis to show approximation guarantees of our models. Experiments on real large-scale mobile social networks show that the proposed methods are much faster than previous algorithms, meanwhile, with high accuracy.