In recent years, Large Language Models (LLMs) have demonstrated remarkable capabilities across various domains.Within the training pipeline of LLMs, the Reinforcement Learning with Human Feedback (RLHF) phase is crucial for aligning LLMs with human preferences and values.Label smoothing, a technique that replaces hard labels with soft labels, emerges as promising techniques to enhance RLHF training.Despite the benefits, the choice of label smoothing parameters often relies on heuristic approaches and lack theoretical understanding.This paper addresses the challenge of selecting the label smoothing parameter in a principled manner.We introduce Confidence Aware Label Smoothing (CALS), a method that iteratively updates the label smoothing parameter based on preference labels and model forecasts.Our theoretical analysis characterizes the optimal label smoothing parameter, demonstrates its dependence on the confidence level, and reveals its influence on training dynamics and equilibrium.Empirical evaluations on state-of-the-art alignment tasks show that CALS achieves competitive performance, highlighting its potential for improving alignment.

Recently an influx of studies claims emergent cognitive abilities in large language models (LLMs). Yet, most rely on anecdotes, overlook contamination of training sets, or lack systematic Evaluation involving multiple tasks, control conditions, multiple iterations, and statistical robustness tests. Here we make two major contributions. First, we propose CogEval, a cognitive science-inspired protocol for the systematic evaluation of cognitive capacities in LLMs. The CogEval protocol can be followed for the evaluation of various abilities. Second, here we follow CogEval to systematically evaluate cognitive maps and planning ability across eight LLMs (OpenAI GPT-4, GPT-3.5-turbo-175B, davinci-003-175B, Google Bard, Cohere-xlarge-52.4B, Anthropic Claude-1-52B, LLaMA-13B, and Alpaca-7B). We base our task prompts on human experiments, which offer both established construct validity for evaluating planning, and are absent from LLM training sets. We find that, while LLMs show apparent competence in a few planning tasks with simpler structures, systematic evaluation reveals striking failure modes in planning tasks, including hallucinations of invalid trajectories and falling in loops. These findings do not support the idea of emergent out-of-the-box planning ability in LLMs. This could be because LLMs do not understand the latent relational structures underlying planning problems, known as cognitive maps, and fail at unrolling goal-directed trajectories based on the underlying structure. Implications for application and future directions are discussed.

Model-free reinforcement learning is known to be memory and computation efficient and more amendable to large scale problems. In this paper, two model-free algorithms are introduced for learning infinite-horizon average-reward Markov Decision Processes (MDPs). The first algorithm reduces the problem to the discounted-reward version and achieves $\mathcal{O}(T^{2/3})$ regret after $T$ steps, under the minimal assumption of weakly communicating MDPs. To our knowledge, this is the first model-free algorithm for general MDPs in this setting. The second algorithm makes use of recent advances in adaptive algorithms for adversarial multi-armed bandits and improves the regret to $\mathcal{O}(\sqrt{T})$, albeit with a stronger ergodic assumption. This result significantly improves over the $\mathcal{O}(T^{3/4})$ regret achieved by the only existing model-free algorithm by Abbasi-Yadkori et al. (2019) for ergodic MDPs in the infinite-horizon average-reward setting.

This paper investigates the sequential opportunistic channel sensing and access problem of an unlicensed or secondary user (SU) in presence of multiple licensed or primary users (PU). The SU senses one channel at a time for the presence of PU. The sensing is considered to be imperfect, i.e, the presence or absence of PU may not be detected correctly. The problem of selection of best channel amongst the sequentially scanned channels for the transmission of SU is modeled as an Optimal Stopping Problem (OSP) which also takes into account the cost of scan per channel. A simple threshold based policy is presented as a solution to OSP in which SU after sensing the channel to be free of PU activity checks the channel quality by comparing the channel gain with a threshold and if the channel gain is more than the threshold then SU selects this channel for transmission. The threshold is seen to be dependent on probabilities of channel occupancy and detection errors. The variation of threshold with respect to detection errors is analyzed and supported with plots. The plots for the variation are presented which match closely with the analytical results.