Existing attacks against multimodal language models (MLLMs) primarily communicate instructions through text accompanied by adversarial images.In contrast, here we exploit the capabilities of MLLMs to interpret non-textual instructions-specifically adversarial images or audio-generated by our novel method, Con Instruction.We optimize the adversarial examples to align closely with target instructions in the embedding space, revealing the detrimental aspects of sophisticated understanding in MLLMs.Unlike previous work, our method does not require training data or preprocessing of textual instructions.While these non-textual adversarial examples can effectively bypass MLLMs safety mechanisms, their combination with various text inputs substantially amplifies attack success.We further introduce a new attack response categorization (ARC) that considers both response quality and relevance to the malicious instructions to evaluate attack success.The results show that Con Instruction effectively bypasses the safety mechanisms in various visual and audio-language models, including LLaVA-v1.5,InternVL, Qwen-VL, and Qwen-Audio, across two standard benchmarks: AdvBench and SafeBench.Specifically, our method achieves the highest attack success rates, reaching 81.3% and 86.6% on LLaVA-v1.5 (13B).On the defense side, we explore various methods against our attacks and find a substantial gap among existing techniques.Our implementation is made available.1Warning: This paper contains examples that may be offensive to some readers.

Although LLM-based conversational agents demonstrate strong fluency and coherence, they still produce undesirable behaviors (errors) that are challenging to prevent from reaching users during deployment.Recent research leverages large language models (LLMs) to detect errors and guide response-generation models toward improvement.However, current LLMs struggle to identify errors not explicitly specified in their instructions, such as those arising from updates to the response-generation model or shifts in user behavior.In this work, we introduce Automated Error Discovery, a framework for detecting and defining errors in conversational AI, and propose SEEED (Soft Clustering Extended Encoder-Based Error Detection), as an encoderbased approach to its implementation.We enhance the Soft Nearest Neighbor Loss by amplifying distance weighting for negative samples and introduce Label-Based Sample Ranking to select highly contrastive examples for better representation learning.SEEED outperforms adapted baselines-including GPT-4o and Phi-4-across multiple error-annotated dialogue datasets, improving the accuracy for detecting unknown errors by up to 8 points and demonstrating strong generalization to unknown intent detection.1 1 We provide our code on GitHub: https://github.com/UKPLab/emnlp2025-automatic-error-discovery.Oh, really?Yes, Rome is an impressive city.I also just came back from summer vacation.I did a

Scientific leaderboards are standardized ranking systems that facilitate evaluating and comparing competitive methods.Typically, a leaderboard is defined by a task, dataset, and evaluation metric (TDM) triple, allowing objective performance assessment and fostering innovation through benchmarking.However, the exponential increase in publications has made it infeasible to construct and maintain these leaderboards manually.Automatic leaderboard construction has emerged as a solution to reduce manual labor.Existing datasets for this task are based on the community-contributed leaderboards without additional curation.Our analysis shows that a large portion of these leaderboards are incomplete, and some of them contain incorrect information.In this work, we present SCILEAD, a manually-curated Scientific Leaderboard dataset that overcomes the aforementioned problems.Building on this dataset, we propose three experimental settings that simulate real-world scenarios where TDM triples are fully defined, partially defined, or undefined during leaderboard construction.While previous research has only explored the first setting, the latter two are more representative of real-world applications.To address these diverse settings, we develop a comprehensive LLM-based framework for constructing leaderboards.Our experiments and analysis reveal that various LLMs often correctly identify TDM triples while struggling to extract result values from publications.We make our code 1 and data 2 publicly available.

In task-oriented dialog (ToD) new intents emerge on regular basis, with a handful of available utterances at best.This renders effective Few-Shot Intent Classification (FSIC) a central challenge for modular ToD systems.Recent FSIC methods appear to be similar: they use pretrained language models (PLMs) to encode utterances and predominantly resort to nearestneighbor-based inference.However, they also differ in major components: they start from different PLMs, use different encoding architectures and utterance similarity functions, and adopt different training regimes.Coupling of these vital components together with the lack of informative ablations prevents the identification of factors that drive the (reported) FSIC performance.We propose a unified framework to evaluate these components along the following key dimensions: (1) Encoding architectures: Cross-Encoder vs Bi-Encoders; (2) Similarity function: Parameterized (i.e., trainable) vs non-parameterized; (3) Training regimes: Episodic meta-learning vs conventional (i.e., non-episodic) training.Our experimental results on seven FSIC benchmarks reveal three new important findings.First, the unexplored combination of cross-encoder architecture and episodic meta-learning consistently yields the best FSIC performance.Second, episodic training substantially outperforms its non-episodic counterpart.Finally, we show that splitting episodes into support and query sets has a limited and inconsistent effect on performance.Our findings show the importance of ablations and fair comparisons in FSIC.We publicly release our code and data 1 .

Unsupervised relation extraction (URE) extracts relations between named entities from raw text without manually-labelled data and existing knowledge bases (KBs).URE methods can be categorised into generative and discriminative approaches, which rely either on hand-crafted features or surface form.However, we demonstrate that by using only named entities to induce relation types, we can outperform existing methods on two popular datasets.We conduct a comparison and evaluation of our findings with other URE techniques, to ascertain the important features in URE.We conclude that entity types provide a strong inductive bias for URE. 1

MOTIVATION: Recent neural approaches on event extraction from text mainly focus on flat events in general domain, while there are less attempts to detect nested and overlapping events. These existing systems are built on given entities and they depend on external syntactic tools. RESULTS: We propose an end-to-end neural nested event extraction model named DeepEventMine that extracts multiple overlapping directed acyclic graph structures from a raw sentence. On the top of the bidirectional encoder representations from transformers model, our model detects nested entities and triggers, roles, nested events and their modifications in an end-to-end manner without any syntactic tools. Our DeepEventMine model achieves the new state-of-the-art performance on seven biomedical nested event extraction tasks. Even when gold entities are unavailable, our model can detect events from raw text with promising performance. AVAILABILITY AND IMPLEMENTATION: Our codes and models to reproduce the results are available at: https://github.com/aistairc/DeepEventMine. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Most deep language understanding models depend only on word representations, which are mainly based on language modelling derived from a large amount of raw text. These models encode distributional knowledge without considering syntactic structural information, although several studies have shown benefits of including such information. Therefore, we propose new syntactically-informed word representations (SIWRs), which allow us to enrich the pre-trained word representations with syntactic information without training language models from scratch. To obtain SIWRs, a graph-based neural model is built on top of either static or contextualised word representations such as GloVe, ELMo and BERT. The model is first pre-trained with only a relatively modest amount of task-independent data that are automatically annotated using existing syntactic tools. SIWRs are then obtained by applying the model to downstream task data and extracting the intermediate word representations. We finally replace word representations in downstream models with SIWRs for applications. We evaluate SIWRs on three information extraction tasks, namely nested named entity recognition (NER), binary and n-ary relation extractions (REs). The results demonstrate that our SIWRs yield performance gains over the base representations in these NLP tasks with 3–9% relative error reduction. Our SIWRs also perform better than fine-tuning BERT in binary RE. We also conduct extensive experiments to analyse the proposed method.

OBJECTIVE: Identification of drugs, associated medication entities, and interactions among them are crucial to prevent unwanted effects of drug therapy, known as adverse drug events. This article describes our participation to the n2c2 shared-task in extracting relations between medication-related entities in electronic health records. MATERIALS AND METHODS: We proposed an ensemble approach for relation extraction and classification between drugs and medication-related entities. We incorporated state-of-the-art named-entity recognition (NER) models based on bidirectional long short-term memory (BiLSTM) networks and conditional random fields (CRF) for end-to-end extraction. We additionally developed separate models for intra- and inter-sentence relation extraction and combined them using an ensemble method. The intra-sentence models rely on bidirectional long short-term memory networks and attention mechanisms and are able to capture dependencies between multiple related pairs in the same sentence. For the inter-sentence relations, we adopted a neural architecture that utilizes the Transformer network to improve performance in longer sequences. RESULTS: Our team ranked third with a micro-averaged F1 score of 94.72% and 87.65% for relation and end-to-end relation extraction, respectively (Tracks 2 and 3). Our ensemble effectively takes advantages from our proposed models. Analysis of the reported results indicated that our proposed approach is more generalizable than the top-performing system, which employs additional training data- and corpus-driven processing techniques. CONCLUSIONS: We proposed a relation extraction system to identify relations between drugs and medication-related entities. The proposed approach is independent of external syntactic tools. Analysis showed that by using latent Drug-Drug interactions we were able to significantly improve the performance of non-Drug-Drug pairs in EHRs.