Large Language Models (LLMs) are often used as automated judges to evaluate text, but their effectiveness can be hindered by various unintentional biases. We propose using linear classifying probes, trained by leveraging differences between contrasting pairs of prompts, to directly access LLMs’ latent knowledge and extract more accurate preferences. Through extensive experiments using models of varying size from four different families and six diverse datasets assessing text quality evaluation and common sense reasoning, we demonstrate that both supervised and unsupervised probing approaches consistently outperform traditional generation-based judgement while maintaining similar computational costs. These probes generalise under domain shifts and can even outperform finetuned evaluators with the same training data size. Our results suggest linear probing offers an accurate, robust and computationally efficient approach for LLM-as-judge tasks while providing interpretable insights into how models encode judgement-relevant knowledge. Our data and code will be openly released in the future.

The deployment of language models brings challenges in generating reliable information, especially when these models are fine-tuned using human preferences.To extract encoded knowledge without (potentially) biased human labels, unsupervised probing techniques like Contrast-Consistent Search (CCS) have been developed (Burns et al., 2022).However, salient but unrelated features in a given dataset can mislead these probes (Farquhar et al., 2023).Addressing this, we propose a cluster normalization method to minimize the impact of such features by clustering and normalizing activations of contrast pairs before applying unsupervised probing techniques.While this approach does not address the issue of differentiating between knowledge in general and simulated knowledge-a major issue in the literature of latent knowledge elicitation (Christiano et al., 2021)-it significantly improves the ability of unsupervised probes to identify the intended knowledge amidst distractions.1 .

According to the Lancet report on global burden of disease published in October 2020, air pollution is amongst the five highest risk factors for global health, reducing life expectancy on average by 20 months. This paper describes a data-driven method for establishing causal relationships between two time-series data streams derived from wearable sensors: personal exposure to airborne particulate matter (PM) of aerodynamic sizes less than 2.5 μm(PM2.5) gathered from the Airspeck monitor, and continuous respiratory rate (breaths/minute) measured by the wireless Respeck monitor worn as a plaster on the chest. Results are presented for a cohort of asthmatic adolescents using the PCMCI method on the short-term causal relationship between PM2.5exposure and respiratory rate for time lags in the first 60 minutes at minute-level intervals, and for time lags between 2 to 8 hours at 10-minute time intervals. For the first time a personalised exposure-response relationship between PM2.5exposure and respiratory rate has been demonstrated for short-term effects in asthmatic adolescents during their every day lives.