This paper studies the spatio-temporal video grounding task, which aims to localize a spatio-temporal tube in an untrimmed video based on the given text description of an event. Existing one-stage approaches suffer from insufficient space-time interaction in two aspects: i) less precise prediction of event temporal boundaries, and ii) inconsistency in object prediction for the same event across adjacent frames. To address these issues, we propose a framework of Comprehensive Space-Time entAnglement (CoSTA) to densely entangle space-time multi-modal features for spatio-temporal localization. Specifically, we propose a space-time collaborative encoder to extract comprehensive video features and leverage Transformer to perform spatio-temporal multi-modal understanding. Our entangled decoder couples temporal boundary prediction and spatial localization via an entangled query, boasting an enhanced ability to capture object-event relationships. We conduct extensive experiments on the challenging benchmarks of HC-STVG and VidSTG, where CoSTA outperforms existing state-of-the-art methods, demonstrating its effectiveness for this task.

The Multi-modal Large Language Model (MLLM) based Referring Expression Generation (REG) task has gained increasing popularity, which aims to generate an unambiguous text description that applies to exactly one object or region in the image by leveraging foundation models. We empirically found that there exists a potential trade-off between the detailedness and the correctness of the descriptions for the referring objects. On the one hand, generating sentences with more details is usually required in order to provide more precise object descriptions. On the other hand, complicated sentences could easily increase the probability of hallucinations. To address this issue, we propose a training-free framework, named ``unleash-then-eliminate'', which first elicits the latent information in the intermediate layers, and then adopts a cycle-consistency-based decoding method to alleviate the production of hallucinations. Furthermore, to reduce the computational load of cycle-consistency-based decoding, we devise a Probing-based Importance Estimation method to statistically estimate the importance weights of intermediate layers within a subset. These importance weights are then incorporated into the decoding process over the entire dataset, intervening in the next token prediction from intermediate layers. Extensive experiments conducted on the RefCOCOg and PHD benchmarks show that our proposed framework could outperform existing methods on both semantic and hallucination-related metrics. Code will be made available in https://github.com/Glupayy/unleash-eliminate.

Referring expression comprehension aims to localize a natural language description in an image. Using location priors to help reduce inaccuracies in cross-modal alignments is the state of the art for CNN-based methods tackling this problem. Recent Transformer-based models cast aside this idea, making the case for steering away from hand-designed components. In this work, we propose LUNA, which uses language as continuing anchors to guide box prediction in a Transformer decoder, and thus show that language-guided location priors can be effectively exploited in a Transformer-based architecture. Our method first initializes an anchor box from the input expression via a small "proto-decoder,'' and then uses this anchor and its refined successors as location guidance in a modified Transformer decoder. At each decoder layer, the anchor box is first used as a query for gathering multi-modal context, and then updated based on the gathered context (producing the next, refined anchor). In the end, a lightweight assessment pathway evaluates the quality of all produced anchors, yielding the final prediction in a dynamic way. This approach allows box decoding to be conditioned on learned anchors, which facilitates accurate grounding, as we shown in the experiments. Our method outperforms existing state-of-the-art methods on the datasets of ReferIt Game, RefCOCO/+/g, and Flickr30K Entities.

The task of ACE Event Detection (ED) often encounters ambiguous and unseen trigger words. Most conventional ED systems exclusively consider the semantic or syntactic patterns as the additional evidence to resolve the problem of the ambiguous and unseen triggers, but rarely consider taking advantages of structured knowledge of the event itself. In this study, we propose Dynamic Word-Trigger-Argument Graph Neural Networks (DWTA-GNN), a novel framework that leverages event structure knowledge to facilitate the two issues simultaneously. In our approach, we utilize words, entities, and event annotations from training to construct an event background graph, which can provide sufficient information of event structure to better disambiguate polysemous triggers and identify unseen triggers. To make full use of the constructed background graph, we further design a knowledge matching module to dynamically match appropriate event structure knowledge and construct a subgraph for each incoming sentence. Besides, an event-selective graph convolution is applied to filter out the noise in the matched knowledge so as to enhance event representation. Experiments on the ACE2005 dataset show that our model achieves competitive performance and advances previous approaches on ambiguous and unseen trigger words, verifying the effectiveness of incorporating event structure knowledge for event detection.

Multi-entity collaborative relationship extraction is an important but challenging task, which has been attracting a lot of interest and poses significant issues in front of systems aimed at natural language understanding. Instead of designing specific models for single relationship extraction tasks, this paper aims to propose a general framework to extract multiple relations among multiple entities in unstructured text by taking advantage of existing models. Based on performing named entity recognition and relation extraction collaboratively, the framework exploits correlations and information propagation among words and relations in a graph network to grasp fundamental features for final classification. The experimental results on two real-world datasets demonstrate that our framework has remarkable applicability and generalizability, and consistently outperforms the strong competitors by a noticeable margin for multi-entity relation extraction.

Existing works for Dialogue Act Recognition (DAR) pay little attention to the imbalanced distribution of the Dialogue Acts (DAs) and exclusively train their models over very fine-grained DAs in one pass, which leads to a limited performance in recognizing low-frequent DAs. To address this issue, we propose a hierarchical label structured network that explicitly introduces coarse-grained DAs to the original fine-grained DAs. A two-pass multi-head attention mechanism is devised to integrate different levels of DA information into the utterance encoding process, thereby utilizing the information learned from the coarse-grained DAs to guide the recognition of the target fine-grained DAs. Besides, a transfer learning over large-scale dialogue datasets is also employed to further boost label representation of the coarse-grained DAs. Extensive experiments show that our model significantly outperforms the state-of-the-art methods, verifying the effectiveness of integrating the hierarchical structure among DAs.

Abstract Massive progress and fruitful results have been achieved since the implementation of the Belt and Road Initiative (B&R). However, the risk of exchange rate between countries along the Belt and Road cannot be ignored. In this paper, we propose a B&R exchange rate index to evaluate the currency risk of countries along the B&R, along with a research on the multiscale features of the B&R exchange rate index and the effective RMB exchange rate index using the EMD algorithm. Then, we propose an integrated forecasting approach. We adopt the EMD algorithm and the Grey Relational Analysis to decompose and reconstruct the B&R exchange rate index and the effective RMB exchange rate index into three subsections with different volatility characteristics. The prediction outcome of the integrated model for the original series equals a sum of the Auto‐regressive Integrated Moving Average model for trend term and the Long Short‐Term Memory model for market fluctuation term as well as noise term. The results of our prediction demonstrate a much better performance of our proposed integrated forecasting model than pure methods. Lastly, several suggestions are put forward to help relevant enterprises effectively avoid and manage the exchange risks.

OBJECTIVE: Evidence is scarce regarding the safety of long-term drug use, especially for drugs treating chronic diseases. To bridge this knowledge gap, this research investigated the differences in drug exposure between clinical trials and clinical practice. MATERIALS AND METHODS: We extracted drug follow-up times from clinical trials in ClinicalTrials.gov and compared the difference between clinical trials and real-world usage data for 914 drugs taken by 96 645 927 patients. RESULTS: A total of 17.5% of drugs had longer median exposure in practice than in trials, 6% of patients had extended exposure to at least 1 drug, and drugs treating nervous system disorders and cardiovascular diseases were the most common among drugs with high rates of extended exposure. CONCLUSIONS: For most of patients, the drug use length is shorter than the tested length in clinical trials. Still, a remarkable number of patients experienced extended drug exposure, particularly for drugs treating nervous system disorders or cardiovascular disorders.

Paraphrase generation aims to rewrite a text with different words while keeping the same meaning. Previous work performs the task based solely on the given dataset while ignoring the availability of external linguistic knowledge. However, it is intuitive that a model can generate more expressive and diverse paraphrase with the help of such knowledge. To fill this gap, we propose Knowledge-Enhanced Paraphrase Network (KEPN), a transformer-based framework that can leverage external linguistic knowledge to facilitate paraphrase generation. (1) The model integrates synonym information from the external linguistic knowledge into the paraphrase generator, which is used to guide the decision on whether to generate a new word or replace it with a synonym. (2) To locate the synonym pairs more accurately, we adopt an incremental encoding scheme to incorporate position information of each synonym. Besides, a multi-task architecture is designed to help the framework jointly learn the selection of synonym pairs and the generation of expressive paraphrase. Experimental results on both English and Chinese datasets show that our method significantly outperforms the state-of-the-art approaches in terms of both automatic and human evaluation.

We study the commonsense inference task that aims to reason and generate the causes and effects of a given event. Existing neural methods focus more on understanding and representing the event itself, but pay little attention to the relations between different commonsense dimensions (e.g. causes or effects) of the event, making the generated results logically inconsistent and unreasonable. To alleviate this issue, we propose Chain Transformer, a logic enhanced commonsense inference model that combines both direct and indirect inferences to construct a logical chain so as to reason in a more logically consistent way. First, we apply a self-attention based encoder to represent and encode the given event. Then a chain of decoders is implemented to reason and generate for different dimensions following the logical chain, where an attention module is designed to link different decoders and to make each decoder attend to the previous reasoned inferences. Experiments on two real-world datasets show that Chain Transformer outperforms previous methods on both automatic and human evaluation, and demonstrate that Chain Transformer can generate more reasonable and logically consistent inference results.

Dialogue Act Recognition (DAR) is a challenging problem in Natural Language Understanding, which aims to attach Dialogue Act (DA) labels to each utterance in a conversation.However, previous studies cannot fully recognize the specific expressions given by users due to the informality and diversity of natural language expressions.To solve this problem, we propose a Heterogeneous User History (HUH) graph convolution network, which utilizes the user's historical answers grouped by DA labels as additional clues to recognize the DA label of utterances.To handle the noise caused by introducing the user's historical answers, we design sets of denoising mechanisms, including a History Selection process, a Similarity Re-weighting process, and an Edge Re-weighting process.We evaluate the proposed method on two benchmark datasets MSDialog and MRDA.The experimental results verify the effectiveness of integrating user's historical answers, and show that our proposed model outperforms the state-of-the-art methods.

We study the problem of infobox-to-text generation that aims to generate a textual description from a key-value table. Representing the input infobox as a sequence, previous neural methods using end-to-end models without order-planning suffer from the problems of incoherence and inadaptability to disordered input. Recent planning-based models only implement static order-planning to guide the generation, which may cause error propagation between planning and generation. To address these issues, we propose a Tree-like PLanning based Attention Network (Tree-PLAN) which leverages both static order-planning and dynamic tuning to guide the generation. A novel tree-like tuning encoder is designed to dynamically tune the static order-plan for better planning by merging the most relevant attributes together layer by layer. Experiments conducted on two datasets show that our model outperforms previous methods on both automatic and human evaluation, and demonstrate that our model has better adaptability to disordered input.

Generating a textual description from a set of RDF triplets is a challenging task in natural language generation. Recent neural methods have become the mainstream for this task, which often generate sentences from scratch. However, due to the huge gap between the structured input and the unstructured output, the input triples alone are insufficient to decide an expressive and specific description. In this paper, we propose a novel anchor-to-prototype framework to bridge the gap between structured RDF triples and natural text. The model retrieves a set of prototype descriptions from the training data and extracts writing patterns from them to guide the generation process. Furthermore, to make a more precise use of the retrieved prototypes, we employ a triple anchor that aligns the input triples into groups so as to better match the prototypes. Experimental results on both English and Chinese datasets show that our method significantly outperforms the state-of-the-art baselines in terms of both automatic and manual evaluation, demonstrating the benefit of learning guidance from retrieved prototypes to facilitate triple-to-text generation.

OBJECTIVE: Coordination ellipsis is a linguistic phenomenon abound in medical text and is challenging for concept normalization because of difficulty in recognizing elliptical expressions referencing 2 or more entities accurately. To resolve this bottleneck, we aim to contribute a generalizable method to reconstruct concepts from medical coordinated elliptical expressions in a variety of biomedical corpora. MATERIALS AND METHODS: We proposed a graph-based representation model and built a pipeline to reconstruct concepts from coordinated elliptical expressions in medical text (RECEEM). There are 4 modules: (1) identify all possible candidate conjunct pairs from original coordinated elliptical expressions, (2) calculate coefficients for candidate conjuncts using the embedding model, (3) select the most appropriate decompositions by global optimization, and (4) rebuild concepts based on a pathfinding algorithm. We evaluated the pipeline's performance on 2658 coordinated elliptical expressions from 3 different medical corpora (ie, biomedical literature, clinical narratives, and eligibility criteria from clinical trials). Precision, recall, and F1 score were calculated. RESULTS: The F1 scores for biomedical publications, clinical narratives, and research eligibility criteria were 0.862, 0.721, and 0.870, respectively. RECEEM outperformed 2 previously released methods. By incorporating RECEEM into 2 existing NLP tools, the F1 scores increased from 0.248 to 0.460 and from 0.287 to 0.630 on concept mapping of 1125 coordination ellipses. CONCLUSIONS: RECEEM improves concept normalization for medical coordinated elliptical expressions in a variety of biomedical corpora. It outperformed existing methods and significantly enhanced the performance of 2 notable NLP systems for mapping coordination ellipses in the evaluation. The algorithm is open sourced online (https://github.com/chiyuan1126/RECEEM).

Chinese relation extraction is conducted using neural networks with either character-based or word-based inputs, and most existing methods typically suffer from segmentation errors and ambiguity of polysemy.To address the issues, we propose a multi-grained lattice framework (MG lattice) for Chinese relation extraction to take advantage of multi-grained language information and external linguistic knowledge.In this framework, (1) we incorporate word-level information into character sequence inputs so that segmentation errors can be avoided.(2) We also model multiple senses of polysemous words with the help of external linguistic knowledge, so as to alleviate polysemy ambiguity.Experiments on three realworld datasets in distinct domains show consistent and significant superiority and robustness of our model, as compared with other baselines.The source code of this paper can be obtained from https://github.com/ thunlp/Chinese_NRE.

Ning Ding, Ziran Li, Zhiyuan Liu, Haitao Zheng, Zibo Lin. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

OBJECTIVE: Cohort definition is a bottleneck for conducting clinical research and depends on subjective decisions by domain experts. Data-driven cohort definition is appealing but requires substantial knowledge of terminologies and clinical data models. Criteria2Query is a natural language interface that facilitates human-computer collaboration for cohort definition and execution using clinical databases. MATERIALS AND METHODS: Criteria2Query uses a hybrid information extraction pipeline combining machine learning and rule-based methods to systematically parse eligibility criteria text, transforms it first into a structured criteria representation and next into sharable and executable clinical data queries represented as SQL queries conforming to the OMOP Common Data Model. Users can interactively review, refine, and execute queries in the ATLAS web application. To test effectiveness, we evaluated 125 criteria across different disease domains from ClinicalTrials.gov and 52 user-entered criteria. We evaluated F1 score and accuracy against 2 domain experts and calculated the average computation time for fully automated query formulation. We conducted an anonymous survey evaluating usability. RESULTS: Criteria2Query achieved 0.795 and 0.805 F1 score for entity recognition and relation extraction, respectively. Accuracies for negation detection, logic detection, entity normalization, and attribute normalization were 0.984, 0.864, 0.514 and 0.793, respectively. Fully automatic query formulation took 1.22 seconds/criterion. More than 80% (11+ of 13) of users would use Criteria2Query in their future cohort definition tasks. CONCLUSIONS: We contribute a novel natural language interface to clinical databases. It is open source and supports fully automated and interactive modes for autonomous data-driven cohort definition by researchers with minimal human effort. We demonstrate its promising user friendliness and usability.