This study explores the proactive ability of LLMs to seek user support.We propose metrics to evaluate the trade-off between performance improvements and user burden, and investigate whether LLMs can determine when to request help under varying information availability.Our experiments show that without external feedback, many LLMs struggle to recognize their need for user support.The findings highlight the importance of external signals and provide insights for future research on improving support-seeking strategies.

Recent works have shown that large language model (LLM) agents are able to improve themselves from experience, which is an important ability for continuous enhancement post-deployment. However, existing benchmarks primarily evaluate their innate capabilities and do not assess their ability to improve over time. To address this gap, we introduce StreamBench, a pioneering benchmark designed to evaluate the continuous improvement of LLM agents over an input-feedback sequence. StreamBench simulates an online learning environment where LLMs receive a continuous flow of feedback stream and iteratively enhance their performance. In addition, we propose several simple yet effective baselines for improving LLMs on StreamBench, and provide a comprehensive analysis to identify critical components that contribute to successful streaming strategies. Our work serves as a stepping stone towards developing effective online learning strategies for LLMs, paving the way for more adaptive AI systems in streaming scenarios.

In this paper, we address the hallucination problem commonly found in natural language generation tasks.Language models often generate fluent and convincing content but lack consistency with the provided source, resulting in potential inaccuracies.We propose a new decoding method called Fidelity-Enriched Contrastive Search (FECS), which augments the Contrastive Search framework with contextaware regularization terms.FECS promotes tokens that are semantically similar to the provided source while penalizing repetitiveness in the generated text.We demonstrate its effectiveness across two tasks prone to hallucination: abstractive summarization and dialogue generation.Results show that FECS consistently enhances faithfulness across various language model sizes while maintaining output diversity comparable to well-performing decoding algorithms. 1

Large language models (LLMs) have exhibited striking in-context learning (ICL) ability to adapt to target tasks with a few inputoutput demonstrations.For better ICL, different methods are proposed to select representative demonstrations from existing training corpora.However, such settings are not aligned with real-world practices, as end-users usually query LMs without access to demonstration pools.In this work, we introduce SELF-ICL-a simple framework which bootstraps LMs' intrinsic capabilities to perform zero-shot ICL.Given a test input, SELF-ICL first prompts the model to generate pseudoinputs.Next, the model predicts pseudo-labels for the pseudo-inputs via zero-shot prompting.Finally, we perform ICL for the test input with the pseudo-input-label pairs as demonstrations.Evaluation on 23 BIG-Bench Hard tasks shows SELF-ICL outperforms zero-shot baselines on both average accuracy and head-to-head comparison.Moreover, with zero-shot chain-ofthought, SELF-ICL achieves results comparable to using real demonstrations.Additionally, we conduct a range of analyses to validate SELF-ICL's effectiveness and provide insights for its behaviors under different settings.1 * Equal contribution. 1 https://github.com/ntunlplab/Self-ICLFollowing is an example instance for the task: Evaluate the result of a random Boolean expression.Please come up with 3 new, diverse, and creative instances for the task.Example instance: Q: not ( True ) and ( True ) is New instance 1: Q: ( False ) or ( False ) and ( True ) is New instance 2: Q: ( True ) and ( False ) or ( False ) is New instance 3: Q: not ( False ) and ( True ) or ( False ) Task description: Evaluate the result of a random Boolean expression.Q: ( False ) or ( False ) and ( True ) is A: False Q: ( True ) and ( False ) or ( False ) is A: False Q: not ( False ) and ( True ) or ( False ) A: True Q: not ( True ) and ( True ) is

Terrorists tend to coordinate multiple raids which, combined with diversionary attacks are designed to divert the attention of the security forces, causing them to take a longer time to respond. This in turn enables the terrorists to cause more casualties at their primary target. It is important to efficiently reallocate the available security forces to meet these terrorist attacks, especially when the available resources are limited. In this paper we proposed two game theory-based models for the reallocation of security forces for the purpose of countering both primary and diversionary attacks in an urban environment. The first model is used each target and the “attack on hidden-object game” in which one analyzes the interaction behavior between the security force commander and the terrorists within a zero-sum game. The probability of a primary attack to this target is derived from the mixed strategy Nash equilibrium. The second model uses all these probabilities to compute the Shapley value for each target, in terms of the majority of all probabilities for primary attacks. The Shapley values are then used to create a reallocated set of the limited security forces. Experimental results show the approach proposed in this study is more efficient than the traditional method for dealing with ever more likely gunshot events.

Efficient emergency management must determine how and when to alert and advise the critical and appropriate response units to the danger of terrorist attacks, particularly when available resources are limited. We propose a framework for homeland security advisory system that incorporates two game theory models designed to advise response units and raise the alarm. In the first scheme the interactive behaviors between the elements or participants of the multi-emergency response system and the zone response unit are modeled and analyzed as a non-cooperative game, after which the terrorist threat value is derived from the mixed strategy Nash equilibrium. In the second step the value TTV is used to compute the Shapley value of all response units for five different threat levels; fair response unit allocation based on the Shapley value creates a minimum set of SAS deployment costs. Numerical examples show that the emergency manager can use this framework to quantitatively evaluate the terrorist threat to each response unit and easily discover where response units are most at risk within the five threat levels.