The detection of listening effort or cognitive load (CL) has been a major research challenge in recent years. Most conventional techniques utilise physiological or audio-visual sensors and are privacy-invasive and computationally complex. The challenges of synchronization, data alignment and accessibility limitations potentially increase the noise and error probability, compromising the accuracy of CL estimates. This innovative work presents a multi-modal, non-invasive and privacy-preserving approach that combines Radio Frequency (RF) and pupillometry sensing to address these challenges. Custom RF sensors are first designed and developed to capture blood flow changes in specific brain regions with high spatial resolution. Next, multi-modal fusion with pupillometry sensing is proposed and shown to offer a robust assessment of cognitive and listening effort through pupil size and pupil dilation. Our novel approach evaluates RF sensing to estimate CL from cerebral blood flow variations utilizing pupillometry as a baseline. A first-of-its-kind, multi-modal dataset is collected as a new benchmark resource in a controlled environment with participants to comprehend target speech with varying background noise levels. The framework is statistically evaluated using intraclass correlation for pupillometry data (average ICC> 0.95). The correlation between pupillometry and RF data is established through Pearson's correlation (average PCC> 0.79). Further, CL is classified into high and low categories based on RF data using K-means clustering. Future work involves integrating RF sensors with glasses to estimate listening effort for hearing-aid users and utilising RF measurements to optimize speech enhancement based on individual's listening effort and complexity of acoustic environment.

We present the first mechanistic evidence that model-free reinforcement learning agents can learn to plan. This is achieved by applying a methodology based on concept-based interpretability to a model-free agent in Sokoban -- a commonly used benchmark for studying planning. Specifically, we demonstrate that DRC, a generic model-free agent introduced by [Guez et al. (2019)](https://arxiv.org/abs/1901.03559), uses learned concept representations to internally formulate plans that both predict the long-term effects of actions on the environment and influence action selection. Our methodology involves: (1) probing for planning-relevant concepts, (2) investigating plan formation within the agent's representations, and (3) verifying that discovered plans (in the agent's representations) have a causal effect on the agent's behavior through interventions. We also show that the emergence of these plans coincides with the emergence of a planning-like property: the ability to benefit from additional test-time compute. Finally, we perform a qualitative analysis of the planning algorithm learned by the agent and discover a strong resemblance to parallelized bidirectional search. Our findings advance understanding of the internal mechanisms underlying planning behavior in agents, which is important given the recent trend of emergent planning and reasoning capabilities in LLMs through RL.

Cognitive load (CL) refers to the mental effort required to process information. Monitoring increased CL is important as it can indicate the onset of cognitive decline and neurodegeneration, allowing for early intervention and management. Traditional techniques using bio-physiological and audio-visual sensors are privacy-invasive and computationally complex. The synchronization, data alignment, and accessibility problems with these techniques can lead to increased noise and errors, reducing the accuracy of CL estimates. This paper presents a first-of-its-kind Radio Frequency (RF) based sensing system that effectively monitors and classifies cognitive load states by detecting cerebral blood flow variations through backscattered RF signal strength. The RF sensors are designed and miniaturized using microwave computational software and fabricated sensors are integrated with glasses to estimate in-vivo CL variations with on-edge processing and classification. The system is validated through user-oriented audio-only (AO) and audio-visual (AV) trials. Participants are tested on their ability to comprehend target speech with different levels of background noise, assessing the impact on CL. The statistical features from the RF reflection data are processed using machine learning (ML) and deep learning (DL) algorithms implemented on a Raspberry Pi. The participants’ CL is classified into high, medium and low categories separately for AO and AV trials. The Multilayer Perceptron (MLP) achieves an overall accuracy of 85% for AO trials and 66.2% for AV trials, with average training and testing times of 6 seconds and 0.001 seconds, respectively. The promising results suggest that the system is an effective, portable, and low-cost alternative for on-edge CL estimation.

Reinforcement learning from human feedback (RLHF) is a standard approach for fine-tuning large language models to follow instructions. As part of this process, learned reward models are used to approximately model human preferences. However, as imperfect representations of the “true” reward, these learned reward models are susceptible to overoptimization. Gao et al. (2023) studied this phenomenon in a synthetic human feedback setup with a significantly larger “gold” reward model acting as the true reward (instead of humans) and showed that overoptimization remains a persistent problem regardless of the size of the proxy reward model and training data used. Using a similar setup, we conduct a systematic study to evaluate the efficacy of using ensemble-based conservative optimization objectives, specifically worst-case optimization (WCO) and uncertainty-weighted optimization (UWO), for mitigating reward model overoptimization when using two optimization methods: (a) best-of-n sampling (BoN) (b) proximal policy optimization (PPO). We additionally extend the setup of Gao et al. (2023) to include 25% label noise to better mirror real-world conditions. Both with and without label noise we find that conservative optimization practically eliminates overoptimization and improves performance by up to 70% for BoN sampling. For PPO, ensemble-based conservative optimization always reduces overoptimization and outperforms single reward model optimization. Moreover, combining it with a small KL penalty successfully prevents overoptimization at no performance cost. Overall, our results demonstrate that ensemble-based conservative optimization can effectively counter overoptimization.

Vascular dementia is the second most common form of dementia prevalent in old age groups, and is also one of the leading causes of mortality. Timely diagnosis and detection of vascular dementia is critical to avoid brain damage. Brain imaging is an essential tool for diagnosis and determines future treatment options available to the patient. Currently, Magnetic Resonance Imaging (MRI), Positron Emission Tomography (PET), Computerized Tomography (CT) scan and carotid ultrasound are mainly used for brain imaging and vascular dementia diagnosis. However, these technologies are expensive, require extensive medical supervision and are not easily accessible. This can cause substantial delay in diagnosis and potentially lead to irreversible damage. This paper presents a first-of-its-kind, miniaturized octagonal monopole-patch antenna (OMPA) sensor for vascular dementia detection. It is designed to operate as part of a portable device and can effectively diagnose underlying causes like brain infarction, stroke and blood clots at the initial stage. The developed sensor designs are validated using microwave computational software and fabricated models are experimentally verified using artificial stroke and blood clot targets inside an artificial brain model. Simulated and measured reflection coefficient results are consistent, and target objects are detected successfully. The findings show that the prototype device is viable as an efficient, portable and low-cost alternate for vascular dementia detection.

In real world settings, numerous constraints are present which are hard to specify mathematically. However, for the real world deployment of reinforcement learning (RL), it is critical that RL agents are aware of these constraints, so that they can act safely. In this work, we consider the problem of learning constraints from demonstrations of a constraint-abiding agent’s behavior. We experimentally validate our approach and show that our framework can successfully learn the most likely constraints that the agent respects. We further show that these learned constraints are \textit{transferable} to new agents that may have different morphologies and/or reward functions. Previous works in this regard have either mainly been restricted to tabular (discrete) settings, specific types of constraints or assume the environment’s transition dynamics. In contrast, our framework is able to learn arbitrary \textit{Markovian} constraints in high-dimensions in a completely model-free setting. The code is available at: \url{https://github.com/shehryar-malik/icrl}.