Kinjal Basu, Ibrahim Abdelaziz, Kiran Kate, Mayank Agarwal, Maxwell Crouse, Yara Rizk, Kelsey Bradford, Asim Munawar, Sadhana Kumaravel, Saurabh Goyal, Xin Wang, Luis A. Lastras, Pavan Kapanipathi. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

There are increasingly many large language models (LLMs) available to the public. While these LLMs have exhibited impressive abilities on a variety of task, any individual LLM in particular may do well on some tasks and worse on others. Additionally, the performance of these models is heavily dependent on the choice of prompt template used. For instance, they exhibit sensitivity to the few shot examples chosen or brittleness to the wording of instructions. Moreover, a prompt template that makes a model perform well for one input may not be the optimal template for another input. This necessitates an approach for adaptively selecting LLM and prompt template pairs for each input. Recent work has shown that the accuracy of LLM's responses is correlated with the LLM's confidence in the response. Thus, a natural choice for selecting which model and prompt template to use is to select the pair that is most confident in its response. However, existing confidence metrics are expensive to calculate - necessitating multiple calls to each LLm and prompt pair. We thus propose an approach to predict the confidence of each pair using an auxiliary regression model that is inexpensive to run. Using this auxiliary model, we select the LLM and prompt template with the highest predicted confidence for a given input. Results on a range of benchmark datasets show that our confidence-based instance-level prompt search method consistently improves the performance of LLMs.

The popularity of conversational digital assistants has resulted in the availability of large amounts of conversational data which can be utilized for improved user experience and personalized response generation.Building these assistants using popular large language models like ChatGPT also require additional emphasis on prompt engineering and evaluation methods.Textual similarity metrics are a key ingredient for such analysis and evaluations.While many similarity metrics have been proposed in the literature, they have not proven effective for taskoriented conversations as they do not take advantage of unique conversational features.To address this gap, we present TaskDiff, a novel conversational similarity metric that utilizes different dialogue components (utterances, intents, and slots) and their distributions to compute similarity.Extensive experimental evaluation of TaskDiff on a benchmark dataset demonstrates its superior performance and improved robustness over other related approaches.

Bright spots, strong indicators of the existence of hydrocarbon accumulations, have been primarily used by geophysicists in oil and gas exploration. Recently, machine-learning algorithms, adopted to automate bright spot detection, have mainly relied on feature extraction and shallow classification workflows to achieve an 85.4% F1 score at best, on 2-D seismic data. Deep neural networks have proved their effectiveness in image classification applications, outperforming humans in some instances, but have not been applied to bright spot detection yet. However, their data-hungry nature poses a challenge in domains suffering from expensive data acquisition, such as seismic data analysis problems; they generally require millions of training samples before achieving good performance. In this article, we implement SeisNet, a convolutional neural network with a “butterfly” architecture that overcame the limited data challenge by implementing data augmentation and inductive transfer-learning techniques. Moreover, we adopt a novel formulation that allows us to detect bright spots and estimate their volume. Our approach was tested against different pretraining and transfer-learning methods and was shown to outperform other approaches in the literature by achieving a 95.6% F1 score on bright spot detection. Our model accurately predicted the volume of the bright spot with an average absolute error that is not more than 0.04% of the total volume of the seismic image. This article is an important step in establishing pretrained networks for other seismic applications such as earthquake prediction; our domain-specific pretrained network, proven to outperform state-of-the-art pretrained networks on bright spot detection, may be used to jump-start the training of deep models on other seismic problems.

Medical students undergo exams, called "Objective Structured Clinical Examinations" (OSCEs), to assess their medical competence in clinical tasks. In these OSCEs, a medical student interacts with a standardized patient, asking questions to complete a clinical assessment of the patient's medical case. In real OSCEs, standardized patients or "Actors" are recruited and trained to answer questions about symptoms mentioned in a script designed by the medical examiner. Developing a virtual conversational patient for OSCEs would lead to significant logistical savings. In this work, we develop a deep learning framework to improve the virtual patient's conversational skills. First, deep neural networks learned domain specific word embeddings. Then, long short-term memory networks derived sentence embeddings before a convolutional neural network model selected an answer to a given question from a script. Empirical results on a homegrown corpus showed that this framework outperformed other approaches, and reached an accuracy of 81%.

With the “last mile” of the delivery process being the most expensive phase, autonomous package delivery systems are gaining traction as they aim for faster and cheaper delivery of goods to city, urban and rural destinations. This interest is further fueled by the emergence of e-commerce, where many applications can benefit from autonomous package delivery solutions. However, the environment stochasticity, variability and task complexity for autonomous operation make it difficult to deploy such systems in real-world applications without the incorporation of advanced machine learning and optimization algorithms. Moving away from designing a “one size fits all” agent to solve the outdoor package delivery problem and considering ad-hoc teams of agents trained within a data-driven framework could provide the answer. In this work, we propose a delivery scheduling algorithm for heterogeneous multi-agent systems using the pickup and delivery problem (PDP) formulation. Specifically, a 3-index mixed integer program-based PDP that allows coalition formation (PDP-CF) among agents is derived to allow multi-agent PDP schedules. We propose a quantum genetic algorithm to solve for the schedule since it is can better handle the large computational complexity of PDP-CF. Multiple PDP scenario simulations show the merits of the proposed approach.

A principled approach to machine learning (ML) problems because of its mathematical foundations in statistical learning theory, support vector machines (SVM), a non-parametric method, require all the data to be available during the training phase. However, once the model parameters are identified, SVM relies only, for future prediction, on a subset of these training instances, called support vectors (SV). The SVM model is mathematically written as a weighted sum of these SV whose number, rather than the dimensionality of the input space, defines SVM's complexity. Since the final number of these SV can be up to half the size of the training dataset, SVM becomes challenged to run on energy aware computing platforms. We propose in this work Knee-Cut SVM (KCSVM) and Knee-Cut Ordinal Optimization inspired SVM (KCOOSVM) that use a soft trick of ordered kernel values and uniform subsampling to reduce SVM's prediction computational complexity while maintaining an acceptable impact on its generalization capability. When tested on several databases from UCL KCSVM and KCOOSVM produced promising results, comparable to similar published algorithms.

Despite support vector machines' (SVM) robustness and optimality, SVM do not scale well computationally. Suffering from slow training convergence on large datasets, SVM online testing time can be suboptimal because SVM write the classifier hyper-plane model as a sum of support vectors that could total as much as half the datasets. Motivated to speed up SVM real time testing by reducing the number of SV, we introduce in this paper a novel local mixture based SVM (LMSVM) approach that exploits the increased separability provided by the kernel trick, while introducing a onetime computational expense. LMSVM applies kernel k-means clustering to the data in kernel space before pruning unwanted clusters based on a mixture measure for label heterogeneity. LMSVM's computational complexity and classification accuracy on four databases from UCI show promising results and motivate follow on research.