This paper investigates the covertness and reliability of unmanned aerial vehicle (UAV) relay communications assisted by a friendly jammer in the presence of co-channel interference (CCI). In the considered scenario, communication from a source to a destination is relayed by a UAV, while a malicious node attempts to monitor the transmission behaviors of both the source and the UAV. To enhance the system’s covertness, we propose a friendly jammer aided covert communication (FJACC) scheme, which leverages a friendly jammer to degrade the detection capability of the malicious monitoring node by sending artificial noise. We analyze the covertness and reliability of the FJACC scheme by deriving closed-form expressions of the outage probability and detection error probability under CCI, considering that the wireless channels are characterized by Nakagami-mfading for UAV-to-ground links and Rayleigh fading for terrestrial links. Numerical results confirm the superior covertness of the proposed FJACC scheme over traditional non-friendly jammer aided transmission scheme. Furthermore, increasing the number of co-channel interferers or their transmit power exhibits a dual effect: it enhances covertness at the expense of reliability.

Network protocol fuzzing faces significant challenges in exploring cross-state vulnerabilities due to the sparse reward problem in traditional coverage-guided approaches. Existing tools struggle with extremely low success rates that lead to uniform sampling and ineffective state space exploration. This paper presents SCSPFUZZ, a novel learning-based protocol fuzzing framework that employs Thompson Sampling with enhanced reward criteria and a conscious cross-state mutation engine to overcome these fundamental limitations. SCSPFUZZ employs a clustering mechanism based on syntactic similarity and protocolspecific state coverage to maintain a diverse seed pool, which reduces redundant executions and improves exploration. To tackle the sparse-reward problem, a seed-adaptive selection algorithm dynamically refines reward criteria for more effective guidance. Furthermore, a novel cross-state mutation strategy integrates fast seed crossover with an enhanced byte inference mechanism to boost deep state exploration. Together, the seed clustering mechanism, seed-adaptive selection algorithm, and cross-state mutation strategies enable effective state-aware fuzzing for network protocols. Evaluations on 11 real-world implementations from ProFuzzBench show an average 10.4% increase in code coverage and 14% more unique crashes compared to AFLNet, demonstrating its effectiveness.

Addressing the critical need for robust safety in Large Language Models (LLMs), particularly against adversarial attacks and in-distribution errors, we introduce Reinforcement Learning with Backtracking Feedback (RLBF). This framework advances upon prior methods, such as BSAFE, by primarily leveraging a Reinforcement Learning (RL) stage where models learn to dynamically correct their own generation errors. Through RL with critic feedback on the model's live outputs, LLMs are trained to identify and recover from their actual, emergent safety violations by emitting an efficient "backtrack by x tokens" signal, then continuing generation autoregressively. This RL process is crucial for instilling resilience against sophisticated adversarial strategies, including middle filling, Greedy Coordinate Gradient (GCG) attacks, and decoding parameter manipulations. To further support the acquisition of this backtracking capability, we also propose an enhanced Supervised Fine-Tuning (SFT) data generation strategy (BSAFE+). This method improves upon previous data creation techniques by injecting violations into coherent, originally safe text, providing more effective initial training for the backtracking mechanism. Comprehensive empirical evaluations demonstrate that RLBF significantly reduces attack success rates across diverse benchmarks and model scales, achieving superior safety outcomes while critically preserving foundational model utility.

Interoperability is a fundamental challenge for longenvisioned blockchain applications. A mainstream approach is using Trusted Execution Environment (TEE) to support interoperable off-chain execution. However, this incurs multiple TEE configured with non-trivial storage capabilities running on fragile concurrent processing environments, rendering current strategies based on TEE far from being practical. This paper aims to fill this gap and design a practical interoperability mechanism with simplified TEE as the underlying architecture. Specifically, we present IvyCross, a TEE-based framework that achieves lowcost, privacy-preserving, and race-free blockchain interoperability. IvyCross allows running arbitrary smart contracts across heterogeneous blockchains atop two distributed TEE-powered hosts. We design an incentive scheme based on smart contracts to stimulate the honest behavior of two hosts, bypassing the requirement of the number of TEE and large memory need. We examine the conditions to guarantee the uniqueness of Nash Equilibrium via Game Theory. Furthermore, an extended optimistic concurrency control protocol is designed to ensure the correctness of concurrent contracts execution. We formally prove the security of IvyCross in the Universal Composability (UC) framework and implement a prototype atop Bitcoin, Ethereum, and FISCO BOCS. Extensive experimental results on end-to-end performance and concurrency control demonstrate the efficiency and practicality of IvyCross.

Extreme events are ubiquitous in various domains, including finance, meteorology, and network analysis. Nevertheless, modern deep learning techniques exhibit limitations in capturing extreme samples, treating them as outliers. In this paper, we propose Generalized Pareto Generative Adversarial Network (GPGAN), a novel generative model for tackling the generation of extreme values in distributions. Our methodology specifically entails the direct modeling of the multi-dimensional Generalized Pareto Distribution (GPD) using deep generative neural networks. Compared to prior work, GPGAN excels at faithfully representing the dependence structure exhibited by extreme values. To ensure effective generation, we employ an adaptive Generalized Pareto parameter strategy. This strategy dynamically selects appropriate GPD parameters for each generation step, enabling compatibility with user-defined extremeness functions within the multi-dimensional GPD framework. Extensive evaluations on a benchmark precipitation dataset demonstrate the effectiveness of our model.

To improve word representation learning, we propose a probabilistic prior which can be seamlessly integrated with word embedding models. Different from previous methods, word embedding is taken as a probabilistic generative model, and it enables us to impose a prior regularizing word representation learning. The proposed prior not only enhances the representation of embedding vectors but also improves the model's robustness and stability. The structure of the proposed prior is simple and effective, and it can be easily implemented and flexibly plugged in most existing word embedding models. Extensive experiments show the proposed method improves word representation on various tasks.

Private set intersection (PSI) is an important cryptographic primitive with many real-world applications. Delegating PSI computation to the cloud can effectively reduce management and computational costs of data owners, and therefore has received widespread attention from researchers. However, in the existing delegated PSI schemes, it is hard for the data owner to authorize only a part of its outsourced data, and for users to flexibly select a part of the outsourced encrypted data of the data owner for intersection computation. In this paper, we propose the multi-subset fine-grained authorization PSI (MA-PSI) over outsourced encrypted datasets. Specially, our protocol is designed for the multi-subset case where each subset is associated with one single tag for data classification. Through tag classification and encryption technology, the data owner can perform fine-grained authorization on its subsets. By querying tags, the user can flexibly select any subset of the data owner for intersection computation. The security definition of the MA-PSI protocol is given, and its security is proved in the semi-honest model. Experimental results show that our protocol is very efficient and the computational cost is linear with the size of the subset. Specifically, the intersection algorithm SetI in our MA-PSI is about 80× (the subset size is 215) - 2500× (the subset size is 210) faster than APSI (Wang et al., IEEE TIFS 2021).

We study a normalizing flow in the latent space of a top-down generator model, in which the normalizing flow model plays the role of the informative prior model of the generator. We propose to jointly learn the latent space normalizing flow prior model and the top-down generator model by a Markov chain Monte Carlo (MCMC)-based maximum likelihood algorithm, where a short-run Langevin sampling from the intractable posterior distribution is performed to infer the latent variables for each observed example, so that the parameters of the normalizing flow prior and the generator can be updated with the inferred latent variables. We show that, under the scenario of non-convergent short-run MCMC, the finite step Langevin dynamics is a flow-like approximate inference model and the learning objective actually follows the perturbation of the maximum likelihood estimation (MLE). We further point out that the learning framework seeks to (i) match the latent space normalizing flow and the aggregated posterior produced by the short-run Langevin flow, and (ii) bias the model from MLE such that the short-run Langevin flow inference is close to the true posterior. Empirical results of extensive experiments validate the effectiveness of the proposed latent space normalizing flow model in the tasks of image generation, image reconstruction, anomaly detection, supervised image inpainting and unsupervised image recovery.

Lifelong seq2seq language generation models are trained with multiple domains in a lifelong learning manner, with data from each domain being observed in an online fashion. It is a well-known problem that lifelong learning suffers from the catastrophic forgetting (CF). To handle this challenge, existing works have leveraged experience replay or dynamic architecture to consolidate the past knowledge, which however result in incremental memory space or high computational cost. In this work, we propose a novel framework name "power norm based lifelong learning" (PNLLL), which aims to remedy the catastrophic forgetting issues with a power normalization on NLP transformer models. Specifically, PNLLL leverages power norm to achieve a better balance between past experience rehearsal and new knowledge acquisition. These designs enable the knowledge adaptation onto new tasks while memorizing the experience of past tasks. Our experiments on paraphrase generation tasks show that PNLLL not only outperforms SOTA models by a considerable margin and but also largely alleviates forgetting.

Learning disentangled representations is an important topic in machine learning with a wide range of applications. Disentangled latent variables represent interpretable semantic information and reflect separate factors of variation in data. Although generative models can learn latent representations as well, most existing models ignore the structural information among latent variables. In this paper, we propose a novel approach to learn the disentangled latent structural representations from data using decomposable variational auto-encoders. We design a novel message passing prior for the latent representations to capture the interactions among different data components. Different from many previous methods that ignore data component or object interaction, our approach simultaneously learns component representation and encodes component relationships. We have applied our model to tasks of data segmentation and latent representation learning among different data components. Experiments on several benchmarks demonstrate the utility of the proposed method.

Catastrophic forgetting poses a major challenge in continual learning where the old knowledge is forgotten when the model is updated on new tasks. Existing solutions tend to solve this challenge through generative models or exemplar-replay strategies. However, such methods may not alleviate the issue that the low-quality samples are generated or selected for the replay, which would directly reduce the effectiveness of the model, especially in the class imbalance, noise, or redundancy scenarios. Accordingly, how to select a suitable coreset during continual learning becomes significant in such setting. In this work, we propose a novel approach that leverages continual coreset sampling (CCS) to address these challenges. We aim to select the most representative subsets during each iteration. When the model is trained on new tasks, it closely approximates/matches the gradient of both the previous and current tasks with respect to the model parameters. This way, adaptation of the model to new datasets could be more efficient. Furthermore, different from the old data storage for maintaining the old knowledge, our approach choose to preserving them in the latent space. We augment the previous classes in the embedding space as the pseudo sample vectors from the old encoder output, strengthened by the joint training with selected new data. It could avoid data privacy invasions in a real-world application when we update the model. Our experiments validate the effectiveness of our proposed approach over various CV/NLP datasets under against current baselines, and we also indicate the obvious improvement of model adaptation and forgetting reduction in a data-free manner.

Conventional natural language process (NLP) generation models are trained offline with a given dataset for a particular task, which is referred to as isolated learning.Research on sequence-to-sequence language generation aims to study continual learning model to constantly learning from sequentially encountered tasks.However, continual learning studies often suffer from catastrophic forgetting, a persistent challenge for lifelong learning.In this paper, we present a novel NLP transformer model that attempts to mitigate catastrophic forgetting in online continual learning from a new perspective, i.e., attention calibration.We model the attention in the transformer as a calibrated unit in a general formulation, where the attention calibration could give benefits to balance the stability and plasticity of continual learning algorithms through influencing both their forward inference path and backward optimization path.Our empirical experiments, paraphrase generation and dialog response generation, demonstrate that this work outperforms state-of-theart models by a considerable margin and effectively mitigate the forgetting.

Causal effect inference has many applications in data analysis and predictions, e.g., user behavior modeling, medical treatment effect prediction, etc. We introduce a new method to perform causal effect inference using flow-based latent-variable models. Our method leverages the expressive power of flow-based models and tries to recover the complex relationship between observations and unobserved confounders. A methodology has been developed to perform causal effect inference along with theoretical analysis. Experimental studies are presented to verify the proposed approach. Empirical results show that the proposed method outperforms baselines on different datasets.

This paper introduces a novel approach embedding flow-based models in hierarchical structures. The proposed model learns the representation of high-dimensional data via a message-passing scheme by integrating flow-based functions through variational inference. Meanwhile, our model produces a representation of the data using a lower dimension, thus overcoming the drawbacks of many flow-based models, usually requiring a high dimensional latent space involving many trivial variables. With the proposed aggregation nodes, our model provides a new approach for distribution modeling and numerical inference on datasets. Multiple experiments on synthetic and real-world datasets show the benefits of our~proposed~method and potentially broad applications.

Dingcheng Li, Zheng Chen, Eunah Cho, Jie Hao, Xiaohu Liu, Fan Xing, Chenlei Guo, Yang Liu. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

Paraphrase generation is an important language generation task attempting to interpret user intents and systematically generate new phrases of identical meanings to the given ones.However, the effectiveness of paraphrase generation is constrained by the access to the golden labeled data pairs where both the amount and the quality of the training data pairs are restricted.In this paper, we propose a new weakly supervised paraphrase generation approach that extends the success of a recent work that leverages reinforcement learning (RL) for effective model training with data selection.While data selection is privileged for the target task which has noisy data, developing a reinforced selective learning regime faces several unresolved challenges.In this paper, we carry on important discussions about the above problem and present a new model that could partially overcome the discussed issues with a model-based planning feature and a reward normalization feature.We perform extensive evaluation on four weakly supervised paraphrase generation tasks where the results show that our method could significantly improve the state-of-the-art performance on the evaluation datasets.

Paraphrase generation is a longstanding NLP task that has diverse applications for downstream NLP tasks.However, the effectiveness of existing efforts predominantly relies on large amounts of golden labeled data.Though unsupervised endeavors have been proposed to address this issue, they may fail to generate meaningful paraphrases due to the lack of supervision signals.In this work, we go beyond the existing paradigms and propose a novel approach to generate high-quality paraphrases with weak supervision data.Specifically, we tackle the weakly-supervised paraphrase generation problem by: (1) obtaining abundant weakly-labeled parallel sentences via retrievalbased pseudo paraphrase expansion; and (2) developing a meta-learning framework to progressively select valuable samples for finetuning a pre-trained language model, i.e., BART, on the sentential paraphrasing task.We demonstrate that our approach achieves significant improvements over existing unsupervised approaches, and is even comparable in performance with supervised state-of-the-arts.

For voice assistants like Alexa, Google Assistant and Siri, correctly interpreting users' intentions is of utmost importance.However, users sometimes experience friction with these assistants, caused by errors from different system components or user errors such as slips of the tongue.Users tend to rephrase their query until they get a satisfactory response.Rephrase detection is used to identify the rephrases and has long been treated as a task with pairwise input, which does not fully utilize the contextual information (e.g.users' implicit feedback).To this end, we propose a contextual rephrase detection model ContReph to automatically identify rephrases from multiturn dialogues.We showcase how to leverage the dialogue context and user-agent interaction signals, including user's implicit feedback and the time gap between different turns, which can help significantly outperform the pairwise rephrase detection models.

Multi-label learning algorithms have attracted more and more attention as of recent. This is mainly because real-world data is generally associated with multiple and non-exclusive labels, which could correspond to different objects, scenes, actions, and attributes. In this paper, we consider the following challenging multi-label stream scenario: the new labels emerge continuously in the changing environments, and are assigned to the previous data. In this setting, data mining solutions must be able to learn the new concepts and avoid catastrophic forgetting simultaneously. We propose a novel continual and interactive feature distillation-based learning framework (CIFDM), to effectively classify instances with novel labels. We utilize the knowledge from the previous tasks to learn new knowledge to solve the current task. Then, the system compresses historical and novel knowledge and preserves it while waiting for new emerging tasks. CIFDM consists of three components: 1) a knowledge bank that stores the existing feature-level compressed knowledge, and predicts the observed labels so far; 2) a pioneer module that aims to learn and predict new emerged labels based on knowledge bank.; 3) an interactive knowledge compression function which is used to compress and transfer the new knowledge to the bank, and then apply the current compressed knowledge to initialize the label embedding of the pioneer for the next task.

Training generative models that can generate high-quality text with sufficient diversity is an important open problem for Natural Language Generation (NLG) community. Recently, generative adversarial models have been applied extensively on text generation tasks, where the adversarially trained generators alleviate the exposure bias experienced by conventional maximum likelihood approaches and result in promising generation quality. However, due to the notorious defect of mode collapse for adversarial training, the adversarially trained generators face a quality-diversity trade-off, i.e., the generator models tend to sacrifice generation diversity severely for increasing generation quality. In this paper, we propose a novel approach which aims to improve the performance of adversarial text generation via efficiently decelerating mode collapse of the adversarial training. To this end, we introduce a cooperative training paradigm, where a language model is cooperatively trained with the generator and we utilize the language model to efficiently shape the data distribution of the generator against mode collapse. Moreover, instead of engaging the cooperative update for the generator in a principled way, we formulate a meta learning mechanism, where the cooperative update to the generator serves as a high level meta task, with an intuition of ensuring the parameters of the generator after the adversarial update would stay resistant against mode collapse. In the experiment, we demonstrate our proposed approach can efficiently slow down the pace of mode collapse for the adversarial text generators. Overall, our proposed method is able to outperform the baseline approaches with significant margins in terms of both generation quality and diversity in the testified domains.

Text classification is a critical research topic with broad applications in natural language processing.Recently, graph neural networks (GNNs) have received increasing attention in the research community and demonstrated their promising results on this canonical task.Despite the success, their performance could be largely jeopardized in practice since they are: (1) unable to capture high-order interaction between words; (2) inefficient to handle large datasets and new documents.To address those issues, in this paper, we propose a principled model -hypergraph attention networks (HyperGAT), which can obtain more expressive power with less computational consumption for text representation learning.Extensive experiments on various benchmark datasets demonstrate the efficacy of the proposed approach on the text classification task.

In this paper, we propose a reinforcement learning based large scale multi-objective ranking system for optimizing short-video recommendation on an industrial video sharing platform. Multiple competing ranking objective and implicit selection bias in user feedback are the main challenges in real-world platform. In order to address those challenges, we integrate multi-gate mixture of experts and soft actor critic into the ranking system. We demonstrated that our proposed framework can greatly reduce the loss function compared with systems only based on single strategies.

The thriving of deep models and generative models provides approaches to model high dimensional distributions. Generative adversarial networks (GANs) can approximate data distributions and generate data samples from the learned data manifolds as well. In this paper, we propose an approach to estimate the implicit likelihoods of GAN models. A stable inverse function of the generator can be learned with the help of a variance network of the generator. The local variance of the sample distribution can be approximated by the normalized distance in the latent space. Simulation studies and likelihood testing on real-world data sets validate the proposed algorithm, which outperforms several baseline methods in these tasks. The proposed method has been further applied to anomaly detection. Experiments show that the method can achieve state-of-the-art anomaly detection performance on real-world data sets.

Recent neural network models have significantly advanced the task of coreference resolution.However, current neural coreference models are typically trained with heuristic loss functions that are computed over a sequence of local decisions.In this paper, we introduce an end-to-end reinforcement learning based coreference resolution model to directly optimize coreference evaluation metrics.Specifically, we modify the state-of-the-art higherorder mention ranking approach in Lee et al. (2018) to a reinforced policy gradient model by incorporating the reward associated with a sequence of coreference linking actions.Furthermore, we introduce maximum entropy regularization for adequate exploration to prevent the model from prematurely converging to a bad local optimum.Our proposed model achieves new state-of-the-art performance on the English OntoNotes v5.0 benchmark.

Both topic modeling and word embedding map documents onto a low-dimensional space, with the former clustering words into a global topic space and the latter into a local continuous embedding space. In this study, we propose the TMSA framework to unify these two complementary patterns by the construction of a mutual learning mechanism between word-cooccurrence based topic modeling and autoencoder. In our model, word topics generated with topic modeling are passed into auto-encoder to impose topic sparsity so that auto-encoder can learn topic-relevant word representations. In return, word embedding learned by autoencoder is sent back to topic modeling to improve the quality of topic generations. Empirical studies show the effectiveness of the proposed TMSA model in discovering topics and embedding words.

Question answering over knowledge graph (QA-KG) aims to use facts in the knowledge graph (KG) to answer natural language questions. It helps end users more efficiently and more easily access the substantial and valuable knowledge in the KG, without knowing its data structures. QA-KG is a nontrivial problem since capturing the semantic meaning of natural language is difficult for a machine. Meanwhile, many knowledge graph embedding methods have been proposed. The key idea is to represent each predicate/entity as a low-dimensional vector, such that the relation information in the KG could be preserved. The learned vectors could benefit various applications such as KG completion and recommender systems. In this paper, we explore to use them to handle the QA-KG problem. However, this remains a challenging task since a predicate could be expressed in different ways in natural language questions. Also, the ambiguity of entity names and partial names makes the number of possible answers large. To bridge the gap, we propose an effective Knowledge Embedding based Question Answering (KEQA) framework. We focus on answering the most common types of questions, i.e., simple questions, in which each question could be answered by the machine straightforwardly if its single head entity and single predicate are correctly identified. To answer a simple question, instead of inferring its head entity and predicate directly, KEQA targets at jointly recovering the question's head entity, predicate, and tail entity representations in the KG embedding spaces. Based on a carefully-designed joint distance metric, the three learned vectors' closest fact in the KG is returned as the answer. Experiments on a widely-adopted benchmark demonstrate that the proposed KEQA outperforms the state-of-the-art QA-KG methods.

Domain ontology is widely used to index literature for the convenience of literature retrieval. Due to the high cost of manual curation of key aspects from the scientific literature, automated methods are crucially required to assist the process of semantic indexing. However, it is a challenging task due to the huge amount of terms and complex hierarchical relations involved in a domain ontology. In this paper, in order to lessen the curse of dimensionality and enhance the training efficiency, we propose an approach named Deep Level-wise Extreme Multi-label Learning and Classification (Deep Level-wise XMLC), to facilitate the semantic indexing of literatures. Specifically, Deep Level-wise XMLC is composed of two sequential modules. The first module, deep level-wise multi-label learning, decomposes the terms of a domain ontology into multiple levels and builds a special convolutional neural network for each level with category-dependent dynamic max pooling and macro F-measure based weights tuning. The second module, hierarchical pointer generation model merges the level-wise outputs into a final summarized semantic indexing. We demonstrate the effectiveness of Deep Level-wise XMLC by comparing it with several state-of-the-art methods on automatic labeling of MeSH, on literature from PubMed MEDLINE and automatic labeling of AmazonCat13K.

Deep learning models have achieved great successes these days. There are intensive studies of word representation learning for question classification. As questions are typically short texts, existing techniques are often not effective for extracting discriminative representations of questions just from a limited number of words. This motivates us to exploit additional information beyond words in order to improve the representation learning of questions. On one hand, topic modeling often captures meaningful semantic structures from the question corpus. Such global topical information should be helpful for question representations. On the other hand, entities extracted from question themselves provide more auxiliary information for short texts from a local viewpoint. Together with words, topics and entities, question representations can be substantially improved.In this paper, we propose a unified neural network framework by integrating Topic modeling, Word embedding and Entity Embedding (TWEE) for question representation learning. Concretely, we introduce a novel topic sparse autoencoder to incorporate discriminative topics into the representation learning of questions. In addition, both words and entity related information are embedded into the network to help learn a more comprehensive question representation. Empirical experiments show that the proposed TWEE framework outperforms the state-of-the-art methods on different datasets.

Data mapping among different data standards in health institutes is often a necessity when data exchanges occur among different institutes. However, no matter rule-based approaches or traditional machine learning methods, none of these methods have achieved satisfactory results yet. In this work, we propose a deep learning method, mixture feature embedding convolutional neural network (MfeCNN), to convert the data mapping to a multiple classification problem. Multi-modal features were extracted from different semantic space with a medical NLP package and powerful feature embeddings were generated by MfeCNN. Classes as many as ten were classified simultaneously by a fully-connected soft-max layer based on multi-view embedding. Experimental results show that our proposed MfeCNN achieved best results than traditional state-of-the-art machine learning models and also much better results than the convolutional neural network of only using bag-of-words as inputs.

Drug repositioning has been a topic of great attention to researchers and pharmaceutical companies due to its significant impact on the cost of drug discovery. There are several approaches to identify potentially novel drug candidates through repurposing. Literature mining has played a critical role in mining such information from scientific articles. In this paper, we used drug-gene and gene-disease semantic predications extracted from Medline abstracts to generate a list of potential drug-disease pairs. We further ranked the generated pairs, by assigning scores based on the predicates that qualify drug-gene and gene-disease relationships. On comparing the top-ranked drug-disease pairs against the Comparative Toxicogenomics Database (CTD), a curated database for drug-disease relations, we found that a significant percentage of top ranked pairs appeared in CTD. Co-occurrence of these high-ranked pairs in Medline abstracts further improves the confidence in our approach to rank the inferred drug-disease relations higher in the list. Finally, manual evaluation of top ten pairs ranked by our approach revealed that nine of them have some biological significance based on expert judgment.

BACKGROUND: Advances in the next generation sequencing technology has accelerated the pace of individualized medicine (IM), which aims to incorporate genetic/genomic information into medicine. One immediate need in interpreting sequencing data is the assembly of information about genetic variants and their corresponding associations with other entities (e.g., diseases or medications). Even with dedicated effort to capture such information in biological databases, much of this information remains 'locked' in the unstructured text of biomedical publications. There is a substantial lag between the publication and the subsequent abstraction of such information into databases. Multiple text mining systems have been developed, but most of them focus on the sentence level association extraction with performance evaluation based on gold standard text annotations specifically prepared for text mining systems. RESULTS: We developed and evaluated a text mining system, MutD, which extracts protein mutation-disease associations from MEDLINE abstracts by incorporating discourse level analysis, using a benchmark data set extracted from curated database records. MutD achieves an F-measure of 64.3% for reconstructing protein mutation disease associations in curated database records. Discourse level analysis component of MutD contributed to a gain of more than 10% in F-measure when compared against the sentence level association extraction. Our error analysis indicates that 23 of the 64 precision errors are true associations that were not captured by database curators and 68 of the 113 recall errors are caused by the absence of associated disease entities in the abstract. After adjusting for the defects in the curated database, the revised F-measure of MutD in association detection reaches 81.5%. CONCLUSIONS: Our quantitative analysis reveals that MutD can effectively extract protein mutation disease associations when benchmarking based on curated database records. The analysis also demonstrates that incorporating discourse level analysis significantly improved the performance of extracting the protein-mutation-disease association. Future work includes the extension of MutD for full text articles.

As part of the forefront of evidence-based medicine, a systematic review (SR) identifies, appraises, and synthesizes all the available literature relevant to a question of interest in a transparent and systematic way. A time-consuming step in conducting systematic review (SR) is the manual article screening from a list of potentially relevant articles retrieved by librarians. In this study, we propose a multi-level SR supporting framework including three levels: Level 1 - ranking with multiple metrics aiming to assist the screening process by increasing the efficiency without compromising the validity; Level 2 - topic analysis for discovering distributed semantics; and Level 3 - network analysis on relation extracted for comprehensive semantic summarization. The sensitivities in two case studies for relevance ranking reached above 80%, while the screening burdens were lowered to 25% based on a crude relevance ranking approach. Topic analysis based on Latent Dirichlet Allocations (LDAs) showed high consistency among the domain experts' selection of articles. The predicate relation network summarized important predicate relations between medical concepts.

Objective: Point of care access to knowledge from full text journal articles supports decision making and decreases medical errors. However, it is an overwhelming task to obtain full text for all journals and the quality of information for a specific topic such as Congestive Heart Failure (CHF) varies across the journals. In this paper, we develop a method to automatically rate journals for a given clinical topic to enable filtering journals or ranking the articles based on source journal. Materials and Methods: We surveyed 169 cardiologists chosen across the US that practice medicine and publish on the topic of CHF. They provided subjective opinion on how valuable the information provided in 60 journals (chosen by Mayo Clinic cardiologists) is for their clinical decision-making. Results: We identified the top CHF journals chosen by cardiologists and the objective metrics that correlate with their scores. Our best Multiple Linear Regression model has a correlation of 0.880 based on five-fold cross-validation. Discussion: We demonstrated that general journal metrics such as impact factor, h-index and number of articles per year provide better results when used in combination with topic-specific metrics such as number of abstracts indexed with the corresponding MeSH® terms. Conclusion: We obtained a journal priority score formula based on different journal metrics to automatically rate any journal in relation to its perceived importance in CHF. Our study shows that using this formula might yield a reliable list of prioritized journals even across other clinical topics such as Multiple Sclerosis.

Clinical knowledge systems (CKS) facilitate healthcare practitioners to make the best decisions at the point of care by having access to most recent knowledge. However, because of the overabundance of clinical research articles, it is time consuming to add new content manually to any CKS such as UpToDate©and to ensure that it is consistent with evidence. For this reason, many CKS such as Mayo Clinic's AskMayoExpert use expert-based (and not evidence-based) knowledge learned from peer-to-peer communication. The main motivation of this work is to explore methods for assigning references automatically to expert-written content. The system employs information retrieval (IR) techniques to retrieve related sentences from MEDLINE abstracts as evidence candidates and then uses a machine learning model trained with physician's evaluation score on certain journals for re-ranking. Finally, the system utilizes citation counts of individual articles for further refining the ranking. A preliminary evaluation using 59 UpToDate sentences and respective citations as gold standard showed that the median ranking improved 11 folds after adding the journal and citation relevance metrics on the top of baseline that only uses the semantic relevance metric. The system seems promising and ready for trials in real use-case scenarios with experts.

A huge amount of association relationships among biological entities (e.g., diseases, drugs, and genes) are scattered in biomedical literature. How to extract and analyze such heterogeneous data still remains a challenging task for most researchers in the biomedical field. Natural language processing (NLP) has the potential in extracting associations among biological entities from literature. However, association information extracted through NLP can be large, noisy, and redundant which poses significant challenges to biomedical researchers to use such information. To address this challenge, we propose a computational framework to facilitate the use of NLP results. We apply Latent Dirichlet Allocation (LDA) to discover topics based on associations. The networks extracted from each topic provide a disease-specific network for downstream bioinformatics analysis of associations for each topic. We illustrated the framework through the construction of disease-specific networks from Semantic MEDLINE, an NLP-generated association database, followed by the analysis of network properties, such as hub nodes and degree distribution. The results demonstrate that (1) LDA-based approach can group related diseases into the same disease topic; (2) the disease-specific association network follows the scale-free network property, in which hub nodes are enriched in related diseases, genes and drugs.

BACKGROUND: Temporal information detection systems have been developed by the Mayo Clinic for the 2012 i2b2 Natural Language Processing Challenge. OBJECTIVE: To construct automated systems for EVENT/TIMEX3 extraction and temporal link (TLINK) identification from clinical text. MATERIALS AND METHODS: The i2b2 organizers provided 190 annotated discharge summaries as the training set and 120 discharge summaries as the test set. Our Event system used a conditional random field classifier with a variety of features including lexical information, natural language elements, and medical ontology. The TIMEX3 system employed a rule-based method using regular expression pattern match and systematic reasoning to determine normalized values. The TLINK system employed both rule-based reasoning and machine learning. All three systems were built in an Apache Unstructured Information Management Architecture framework. RESULTS: Our TIMEX3 system performed the best (F-measure of 0.900, value accuracy 0.731) among the challenge teams. The Event system produced an F-measure of 0.870, and the TLINK system an F-measure of 0.537. CONCLUSIONS: Our TIMEX3 system demonstrated good capability of regular expression rules to extract and normalize time information. Event and TLINK machine learning systems required well-defined feature sets to perform well. We could also leverage expert knowledge as part of the machine learning features to further improve TLINK identification performance.

OBJECTIVE: This paper describes the coreference resolution system submitted by Mayo Clinic for the 2011 i2b2/VA/Cincinnati shared task Track 1C. The goal of the task was to construct a system that links the markables corresponding to the same entity. MATERIALS AND METHODS: The task organizers provided progress notes and discharge summaries that were annotated with the markables of treatment, problem, test, person, and pronoun. We used a multi-pass sieve algorithm that applies deterministic rules in the order of preciseness and simultaneously gathers information about the entities in the documents. Our system, MedCoref, also uses a state-of-the-art machine learning framework as an alternative to the final, rule-based pronoun resolution sieve. RESULTS: The best system that uses a multi-pass sieve has an overall score of 0.836 (average of B(3), MUC, Blanc, and CEAF F score) for the training set and 0.843 for the test set. DISCUSSION: A supervised machine learning system that typically uses a single function to find coreferents cannot accommodate irregularities encountered in data especially given the insufficient number of examples. On the other hand, a completely deterministic system could lead to a decrease in recall (sensitivity) when the rules are not exhaustive. The sieve-based framework allows one to combine reliable machine learning components with rules designed by experts. CONCLUSION: Using relatively simple rules, part-of-speech information, and semantic type properties, an effective coreference resolution system could be designed. The source code of the system described is available at https://sourceforge.net/projects/ohnlp/files/MedCoref.

OBJECTIVE: To characterise empirical instances of Unified Medical Language System (UMLS) Metathesaurus term strings in a large clinical corpus, and to illustrate what types of term characteristics are generalisable across data sources. DESIGN: Based on the occurrences of UMLS terms in a 51 million document corpus of Mayo Clinic clinical notes, this study computes statistics about the terms' string attributes, source terminologies, semantic types and syntactic categories. Term occurrences in 2010 i2b2/VA text were also mapped; eight example filters were designed from the Mayo-based statistics and applied to i2b2/VA data. RESULTS: For the corpus analysis, negligible numbers of mapped terms in the Mayo corpus had over six words or 55 characters. Of source terminologies in the UMLS, the Consumer Health Vocabulary and Systematized Nomenclature of Medicine-Clinical Terms (SNOMED-CT) had the best coverage in Mayo clinical notes at 106426 and 94788 unique terms, respectively. Of 15 semantic groups in the UMLS, seven groups accounted for 92.08% of term occurrences in Mayo data. Syntactically, over 90% of matched terms were in noun phrases. For the cross-institutional analysis, using five example filters on i2b2/VA data reduces the actual lexicon to 19.13% of the size of the UMLS and only sees a 2% reduction in matched terms. CONCLUSION: The corpus statistics presented here are instructive for building lexicons from the UMLS. Features intrinsic to Metathesaurus terms (well formedness, length and language) generalise easily across clinical institutions, but term frequencies should be adapted with caution. The semantic groups of mapped terms may differ slightly from institution to institution, but they differ greatly when moving to the biomedical literature domain.

Abstract This paper proposes a novel application of topic models to do entity relation detection (ERD). In order to make use of the latent semantics of text, we formulate the task of relation detection as a topic modeling problem. The motivation is to find underlying topics that are indicative of relations between named entities (NEs). Our approach considers pairs of NEs and features associated with them as mini documents, and aims to utilize the underlying topic distributions as indicators for the types of relations that may exist between the NE pair. Our system, ERD-MedLDA, adapts Maximum Entropy Discriminant Latent Dirichlet Allocation (MedLDA) with mixed membership for relation detection. By using supervision, ERD-MedLDA is able to learn topic distributions indicative of relation types. Further, ERD-MedLDA is a topic model that combines the benefits of both, maximum likelihood estimation (MLE) and maximum margin estimation (MME), and the mixed-membership formulation enables the system to incorporate heterogeneous features. We incorporate different features into the system and perform experiments on the ACE 2005 corpus. Our approach achieves better overall performance for precision, recall, and F-measure metrics as compared to baseline SVM-based and LDA-based models. We also find that our system shows better and consistent improvements with the addition of complex informative features as compared to baseline systems.