James Y. Huang, Sailik Sengupta, Daniele Bonadiman, Yi-An Lai, Arshit Gupta, Nikolaos Pappas, Saab Mansour, Katrin Kirchhoff, Dan Roth. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Direct alignment algorithms (DAAs), such as direct preference optimization (DPO), have become popular alternatives to Reinforcement Learning from Human Feedback (RLHF) due to their simplicity, efficiency, and stability. However, the preferences used by DAAs are usually collected before alignment training begins and remain unchanged (off-policy). This design leads to two problems where the policy model (1) picks up on spurious correlations in the dataset (as opposed to only learning alignment to human preferences), and (2) overfits to feedback on off-policy trajectories that have less likelihood of being generated by the updated policy model. To address these issues, we introduce Self-Reviewing and Alignment (SeRA), a cost-efficient and effective method that can be readily combined with existing DAAs. SeRA comprises of two components: (1) sample selection using implicit reward margin to alleviate over-optimization on such undesired features, and (2) preference bootstrapping using implicit rewards to augment preference data with updated policy models in a cost-efficient manner. Extensive experiments, including on instruction-following tasks, demonstrate the effectiveness and generality of SeRA in training LLMs with diverse offline preference datasets and and DAAs.

In scenarios involving robots generating and executing plans, conflicts can arise between cost-effective robot execution and meeting human expectations for safe behavior. When humans supervise robots, their accountability increases, especially when robot behavior deviates from expectations. To address this, robots may choose a highly constrained plan when monitored and a more optimal one when unobserved. While this behavior is not driven by human-like motives, it stems from robots accommodating diverse supervisors. To optimize monitoring costs while ensuring safety, we model this interaction in a trust-based game-theoretic framework. However, pure-strategy Nash equilibrium often fails to exist in this model. To address this, we introduce the concept of a trust boundary within the mixed strategy space, aiding in the discovery of optimal monitoring strategies. Human studies demonstrate the necessity of optimal strategies and the benefits of our suggested approaches.

In this work, we design an Artificially Intelligent Task Allocator (AITA) that proposes a task allocation for a team of humans. A key property of this allocation is that when an agent with imperfect knowledge (about their teammate's costs and/or the team's performance metric) contests the allocation with a counterfactual, a contrastive explanation can always be provided to showcase why the proposed allocation is better than the proposed counterfactual. For this, we consider a negotiation process that produces a negotiation-aware task allocation and, when contested, leverages a negotiation tree to provide a contrastive explanation. With human subject studies, we show that the proposed allocation indeed appears fair to a majority of participants and, when not, the explanations generated are judged as convincing and easy to comprehend.

Yuwei Zhang, Siffi Singh, Sailik Sengupta, Igor Shalyminov, Hang Su, Hwanjun Song, Saab Mansour. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Shamik Roy, Sailik Sengupta, Daniele Bonadiman, Saab Mansour, Arshit Gupta. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

Advances in neural modeling have achieved state-of-the-art (SOTA) results on public natural language processing (NLP) benchmarks, at times surpassing human performance.However, there is a gap between public benchmarks and real-world applications where noise, such as typographical or grammatical mistakes, is abundant and can result in degraded performance.Unfortunately, works which evaluate the robustness of neural models on noisy data and propose improvements, are limited to the English language.Upon analyzing noise in different languages, we observe that noise types vary greatly across languages.Thus, existing investigations do not generalize trivially to multilingual settings.To benchmark the performance of pretrained multilingual language models, we construct noisy datasets covering five languages and four NLP tasks and observe a clear gap in the performance between clean and noisy data in the zero-shot cross-lingual setting.After investigating several ways to boost the robustness of multilingual models in this setting, we propose Robust Contrastive Pretraining (RCP).RCP combines data augmentation with a contrastive loss term at the pretraining stage and achieves large improvements on noisy (& original test data) across two sentencelevel (+3.2%) and two sequence-labeling (+10 F1-score) multilingual classification tasks.Language Noise Injection Ratio Realistic Utt. % Realistic Examples (test-set) Unrealistic Examples (test-set) French (fr) 0.1 95.4% Me montré les vols directs de Charlotte à Minneapolis mardi matin .Quelle compagnie aérienne fut YX Me montré des vols entre Détroit er St. Louis sur Delta Northwest US Air est United Airlines .Lister des vols de Las Vegas à Son Diego German (de) 0.2 94.5% Zeige mir der Flüge zwischen Housten und Orlando Welche Flüge gibt es vom Tacoma nach San Jose Zeige mit alle Flüge vor Charlotte nach Minneapolis zum Dienstag morgen Zeige mit Flüge an Milwaukee nach Washington DC v. 12 Uhr

Automated Planning techniques can be leveraged to build effective decision support systems that assist the human-in-the-loop. Such systems must provide intuitive explanations when the suggestions made by these systems seem inexplicable to the human. In this regard, we consider scenarios where the user questions the system's suggestion by providing alternatives (referred to as foils). In response, we empower existing decision support technologies to engage in an interactive explanatory dialogue with the user and provide contrastive explanations based on user-specified foils to reach a consensus on proposed decisions. To provide contrastive explanations, we adapt existing techniques in Explainable AI Planning (XAIP). Furthermore, we use this dialog to elicit the user's latent preferences and propose three modes of interaction that use these preferences to provide revised plan suggestions. Finally, we showcase a decision support system that provides all these capabilities.

Proactive Decision Support aims at improving the decision making experience of human decision-makers by enhancing the quality of the decisions and the ease of making them. Given that AI techniques are efficient in searching over a potentially large solution space (of decision) and finding good solutions, it can be used for human-in-the-loop scenarios such as disaster response that demand naturalistic decision making. A human decision-maker, in such scenarios, may experience high-cognitive overload leading to a loss of situational awareness. In this paper, we propose the use of automated task-planning techniques coupled with design principles laid out in the Human-Computer Interaction (HCI) community for developing a proactive decision support system. To this extent, we highlight the capabilities of such a system RADAR and briefly, describe how automated planning techniques help us in providing the varying degrees of assistance. To evaluate the effectiveness of the different capabilities, we conduct ablation studies with human subjects on a synthetic environment for making an interactive plan of study. We found that planning techniques like plan validation and suggestions help to reduce planning time (objective metrics) and improves user satisfaction (subjective metrics) compared to expert human planners without any support.

In this brief abstract, we develop a computationally simpler version of the operator count heuristic for a particular class of domains. The contribution of this abstract is thus threefold, we (1) propose an efficient closed form approximation to the operator count heuristic; (2) leverage compressed sensing techniques to obtain an integer approximation in polynomial time; and (3) discuss the relationship of the proposed formulation to existing heuristics and investigate properties of domains where such approaches are useful.