The strong capability of large language models (LLMs) has been applied to information extraction (IE) through either retrieval augmented prompting or instruction tuning (IT). However, the best way to incorporate information with LLMs for IE remains an open question. In this paper, we explore Retrieval Augmented Instruction Tuning (RA-IT) for IE, focusing on the task of open named entity recognition (NER). Specifically, for each training sample, we retrieve semantically similar examples from the training dataset as the context and prepend them to the input of the original instruction. To evaluate our RA-IT approach more thoroughly, we construct a Chinese IT dataset for open NER and evaluate RA-IT in both English and Chinese scenarios. Experimental results verify the effectiveness of RA-IT across various data sizes and in both English and Chinese scenarios. We also conduct thorough studies to explore the impacts of various retrieval strategies in the proposed RA-IT framework.

Semantic information is crucial for human awareness. The ability to extract such information interactively from brain activity using non-invasive technologies like functional Magnetic Resonance Imaging (fMRI) is valuable for medical assistive technologies. However, research in this domain remains relatively limited. To address this gap, we proposes BrainChat, an interactive framework designed to decode semantic information from fMRI. BrainChat leverages a large-scale vision-language model, functioning through fMRI-based captioning and, optionally, question answering. First, a pair of fMRI encoder and decoder is trained to map fMRI data into a latent space representation using Masked Brain Modeling, a self-supervised approach. On the second stage, a projector is added to align these fMRI representations with both pretrained image and text embeddings, yielding a unified representation. A text decoder is also added at this stage, adopting cross-attention with the unified fMRI representation to guide the generation of semantic information. During this stage, the fMRI encoder, the projector, and the text decoder are trained together by minimizing a combined contrastive loss and caption loss. BrainChat achieves state-of-the-art performance in fMRI captioning and implements fMRI question answering, enabling interactive clinical applications. The code is available on Github1.

Existing fMRI-to-image reconstruction methods have made progress in decoding visual images from fMRI but continue to struggle with capturing finer details like shape and color. Inspired by the human brain’s processing of visual information—particularly its encoding at multiple spatial scales across various cortical areas, each with distinct sensitivities to different stimuli, a phenomenon we term Multiscale Encoding and Perceptual Specificity—we propose a method to decode human vision from fMRI data. Specifically, we use a multiscale module to integrate information across different scales, and a channel attention module to focus on features related to the reconstruction task. Notably, our research reveals distinct feature sensitivities in the multiscale module and channel attention module to fMRI data, indicating a potential functional decoupling, as confirmed through experiments. This further guides architectural refinements, as we reorganized the network modules, which led to an improved reconstruction performance. Furthermore, our analysis reveals a correlation between visual stimulus size, fMRI response scale, and various attention scales within the multiscale module. The experimental results help us understand cognitive psychology and neuroscience from the perspective of neural networks. The code is available on GitHub.

With the robust capabilities of recent open Named Entity Recognition (NER) model, Direct Preference Optimization (DPO) through self-training emerges as a promising approach for expanding the application of NER tasks. However, the preference annotations from AI feedback do not fully capture high-quality and fine-grained response in the NER task. To address this issue, we propose SEP, a novel preference learning framework that focuses on Self-Rewarded Entity-Level Preference for the open NER task. Our work employs the NER model itself to automatically annotate entity-level pair for preference optimization. Furthermore, self-rewarding strategy via log probability is explored to score the response, aiming to reduce the impact of low-quality annotations in constructing training data. Based on the response and their reward, multiple entity-level preference pairs via a coarse-to-fine mechanism are constructed to pinpoint detailed wrong entities in response. Experimental results show that SEP achieves significant improvements over vanilla DPO across a benchmark of 20 NER datasets. Ablation study underscores the critical roles of self-generating response rewarding and entity-aware preference learning. Further exploration confirms consistent improvements across multiple iterations and reflects the impact of data size and data domain in the self-training NER task.

With the increasing popularity of golf, more and more amateurs are focusing on this sport. Deep learning based golf swing event detection becomes a meaningful task, yet current research relies on RGB images as the unique input, leading to insufficient accuracy in limited-sample scenarios. In this paper, we propose an improved dual-stream network framework, Frame-Skeleton, for detecting golf swing events. The framework combines RGB images and skeleton sequences, utilizing ST-GCN (Spatial-Temporal Graph Convolutional Network) to process joint features, significantly enhancing recognition accuracy and robustness in scenarios with limited data. Additionally, we construct a golf swing dataset, Frameflow, which contains down-the-line golf swing videos of professional and non-professional golfers, providing a new data source for research. Experimental validation shows that Frame-Skeleton outperforms traditional single-stream methods on the benchmark Golfdb dataset and demonstrates stronger generalization capabilities on the smaller Frameflow dataset.

Graph Databases (Graph DB) find extensive application across diverse domains such as finance, social networks, and medicine. Yet, the translation of Natural Language (NL) into the Graph Query Language (GQL), referred to as NL2GQL, poses significant challenges owing to its intricate and specialized nature. Some approaches have sought to utilize Large Language Models (LLMs) to address analogous tasks like text2SQL. Nonetheless, in the realm of NL2GQL tasks tailored to a particular domain, the absence of domain-specific NL-GQL data pairs adds complexity to aligning LLMs with the graph DB. To tackle this challenge, we present a well-defined pipeline. Initially, we use ChatGPT to generate NL-GQL data pairs, leveraging the provided graph DB and two mutual verification self-instruct methods which ensure consistency between NL and GQL. Subsequently, we employ the generated data to fine-tune LLMs, ensuring alignment between LLMs and the graph DB. Moreover, we find the importance of relevant schema in efficiently generating accurate GQLs. Thus, we introduce a method to extract relevant schema as the input context. We evaluate our method using two carefully constructed datasets derived from graph DBs in the finance and medicine domains, named FinGQL and MediGQL. Experimental results reveal that our approach significantly outperforms a set of baseline methods, with improvements of 5.90 and 6.36 absolute points on EM, and 6.00 and 7.09 absolute points on EX for FinGQL and MediGQL, respectively

Large language models (LLMs) exhibited powerful capability in various natural language processing tasks.This work focuses on exploring LLM performance on zero-shot information extraction, with a focus on the ChatGPT and named entity recognition (NER) task.Inspired by the remarkable reasoning capability of LLM on symbolic and arithmetic reasoning, we adapt the prevalent reasoning methods to NER and propose reasoning strategies tailored for NER.First, we explore a decomposed question-answering paradigm by breaking down the NER task into simpler subproblems by labels.Second, we propose syntactic augmentation to stimulate the model's intermediate thinking in two ways: syntactic prompting, which encourages the model to analyze the syntactic structure itself, and tool augmentation, which provides the model with the syntactic information generated by a parsing tool.Besides, we adapt self-consistency to NER by proposing a two-stage majority voting strategy, which first votes for the most consistent mentions, then the most consistent types.The proposed methods achieve remarkable improvements for zero-shot NER across seven benchmarks, including Chinese and English datasets, and on both domainspecific and general-domain scenarios.In addition, we present a comprehensive analysis of the error types with suggestions for optimization directions.We also verify the effectiveness of the proposed methods on the few-shot setting and other LLMs. 1 * Corresponding authors. 1 Code available at: https://github.com/Emma1066/ Zero-Shot-NER-with-ChatGPT Input Text: The player who temporarily ranks second is German athlete Bao Lizzo, with a total score of 355.02 points, slightly lower than Lanwei.Gold Label: {"German":"Geo-Political Entity", "Lanwei": "Person", "BaoꞏLizzo": "Person"} Vanilla Ans: {"German athlete Bao Lizzo": "Person", "Lanwei": "Person"} TS-SC Ans: {"BaoꞏLizzo": "Person": "Person", "Lanwei": "Person", "German": "Geo-Political Entity"} ----------------------------------

Fault diagnosis in an open world refers to the diagnosis tasks that need to cope with previously unknown faults in the online stage. It faces a great challenge yet to be addressed—that is, the online data of unknown faults may be classified as normal samples with a high probability. In this article, we develop an effective solution for this challenge by using supervised contrastive learning to learn a discriminative and compact embedding for the known normal situation and fault situations. Specifically, in addition to contrasting a given sample with other instances as is the case in conventional contrastive learning methods, our training scheme contrasts the normal samples with negative augmentations of themselves. The negative out-of-distribution data is generated by the Soft Brownian Offset sampling method to simulate the previously unknown faults. Computational experiments are conducted on the Tennessee Eastman Process benchmark dataset and a practical plasma etching process dataset. The proposed method achieves significant improvement compared with four existing methods under three open-set fault diagnosis circumstances, i.e., balanced open-set fault diagnosis, imbalanced fault diagnosis, and few-shot fault diagnosis. This demonstrates its great potentials in real world fault diagnosis applications.

Open Knowledge Graph (OpenKG) link prediction is important for using OpenKGs in applications such as question answering and text comprehension. The noun phrases (NPs) and relation phrases in OpenKGs are not canonicalized, making OpenKG link prediction highly challenging. Existing methods addressing this problem infuse canonicalization information into knowledge graph embedding models. However, they still fail to fully exploit the semantics of NPs. First, two different NPs, even referring to the same entity, can carry different versions of information, which has been ignored by previous methods. Second, neighborhood information of NPs in OpenKGs has not been utilized, which contains abundant information for link prediction. Based on these observations, we propose the OpenKG Segmented Embedding (OKGSE) method. Specifically, to fully capture the dissimilarity of NPs belonging to the same cluster, we learn separate parts of embedding for both the NP cluster and NP. Meanwhile, we exploit neighborhood information by integrating graph context into the semantic matching score function. Extensive experiments across four benchmarks show that OKGSE can achieve state-of-the-art performance as well as effectively capture the unique semantics of each NP.