Large Language Models (LLMs) and causal learning each hold strong potential for clinical decision making (CDM). However, their synergy remains poorly understood, largely due to the lack of systematic benchmarks evaluating their integration in clinical risk prediction. In real-world healthcare, identifying features with causal influence on outcomes is crucial for actionable and trustworthy predictions. While recent work highlights LLMs' emerging causal reasoning abilities, there lacks comprehensive benchmarks to assess their causal learning and performance informed by causal features in clinical risk prediction. To address this, we introduce REACT-LLM, a benchmark designed to evaluate whether combining LLMs with causal features can enhance clinical prognostic performance and potentially outperform traditional machine learning (ML) methods. Unlike existing LLM-clinical benchmarks that often focus on a limited set of outcomes, REACT-LLM evaluates 7 clinical outcomes across 2 real-world datasets, comparing 15 prominent LLMs, 6 traditional ML models, and 3 causal discovery (CD) algorithms. Our findings indicate that while LLMs perform reasonably in clinical prognostics, they have not yet outperformed traditional ML models. Integrating causal features derived from CD algorithms into LLMs offers limited performance gains, primarily due to the strict assumptions of many CD methods, which are often violated in complex clinical data. While the direct integration yields limited improvement, our benchmark reveals a more promising synergy: LLMs serve effectively as knowledge-rich collaborators for identifying and optimizing causal features. Additionally, in-context learning improves LLM predictions when prompts are tailored to the task and model. Different LLMs show varying sensitivity to structured data encoding formats, for example, open-source models perform better with JSON, while smaller models benefit from narrative serialization. These findings highlight the need to match prompts and data formats to model architecture and pretraining.

Fine-tuning large language models (LLMs) improves performance but introduces critical safety vulnerabilities: even minimal harmful data can severely compromise safety measures. We observe that perturbations orthogonal to the alignment direction—defined by weight differences between aligned (safe) and unaligned models—rapidly compromise model safety. In contrast, updates along the alignment direction largely preserve it, revealing the parameter space as a "narrow safety basin". To address this, we propose AsFT (Anchoring Safety in Fine-Tuning) to maintain safety by explicitly constraining update directions during fine-tuning. By penalizing updates orthogonal to the alignment direction, AsFT effectively constrains the model within the "narrow safety basin," thus preserving its inherent safety. Extensive experiments on multiple datasets and models show that AsFT reduces harmful behaviors by up to 7.60%, improves task performance by 3.44%, and consistently outperforms existing methods across multiple tasks.

Multimodal Large Language Models (MLLMs) have emerged to tackle the challenges of Visual Question Answering (VQA), sparking a new research focus on conducting objective evaluations of these models. Existing evaluation methods face limitations due to the significant human workload required to design Q&A pairs for visual images, which inherently restricts the scale and scope of evaluations. Although automated MLLM-as-judge approaches attempt to reduce the human workload through automatic evaluations, they often introduce biases. To address these problems, we propose an Unsupervised Peer review MLLM Evaluation framework. It utilizes only image data, allowing models to automatically generate questions and conduct peer review assessments of answers from other models, effectively alleviating the reliance on human workload. Additionally, we introduce the vision-language scoring system to mitigate the bias issues, which focuses on three aspects: (i) response correctness; (ii) visual understanding and reasoning; and (iii) image-text correlation. Experimental results demonstrate that UPME achieves a Pearson correlation of 0.944 with human evaluations on the MMstar dataset and 0.814 on the ScienceQA dataset, indicating that our framework closely aligns with human-designed benchmarks and inherent human preferences.

Recently, Multimodal Large Language Models (MLLMs) have been used as agents to control keyboard and mouse inputs by directly perceiving the Graphical User Interface (GUI) and generating corresponding commands. However, current agents primarily demonstrate strong understanding capabilities in static environments and are mainly applied to relatively simple domains, such as Web or mobile interfaces. We argue that a robust GUI agent should be capable of perceiving temporal information on the GUI, including dynamic Web content and multi-step tasks. Additionally, it should possess a comprehensive understanding of various GUI scenarios, including desktop software and multi-window interactions. To this end, this paper introduces a new dataset, termed GUI-World, which features meticulously crafted Human-MLLM annotations, extensively covering six GUI scenarios and eight types of GUI-oriented questions in three formats. We evaluate the capabilities of current state-of-the-art MLLMs, including Image LLMs and Video LLMs, in understanding various types of GUI content, especially dynamic and sequential content. Our findings reveal that current models struggle with dynamic GUI content without manually annotated keyframes or operation history. On the other hand, Video LLMs fall short in all GUI-oriented tasks given the sparse GUI video dataset. Therefore, we take the initial step of leveraging a fine-tuned Video LLM, GUI-Vid, as a GUI-oriented assistant, demonstrating an improved understanding of various GUI tasks. However, due to the limitations in the performance of base LLMs, we conclude that using video LLMs as GUI agents remains a significant challenge. We believe our work provides valuable insights for future research in dynamic GUI content understanding. All the dataset and code are publicly available at: https://gui-world.github.io.

Large Language Models (LLMs) such as GPT-4 and Llama3 have significantly impacted various fields by enabling high-quality synthetic data generation and reducing dependence on expensive human-generated datasets. Despite this, challenges remain in the areas of generalization, controllability, diversity, and truthfulness within the existing generative frameworks. To address these challenges, this paper presents DataGen, a comprehensive LLM-powered framework designed to produce diverse, accurate, and highly controllable datasets. DataGen is adaptable, supporting all types of text datasets and enhancing the generative process through innovative mechanisms. To augment data diversity, DataGen incorporates an attribute-guided generation module and a group checking feature. For accuracy, it employs a code-based mathematical assessment for label verification alongside a retrieval-augmented generation technique for factual validation. The framework also allows for user-specified constraints, enabling customization of the data generation process to suit particular requirements. Extensive experiments demonstrate the superior quality of data generated by DataGen, and each module within DataGen plays a critical role in this enhancement. Additionally, DataGen is applied in two practical scenarios: benchmarking LLMs and data augmentation. The results indicate that DataGen effectively supports dynamic and evolving benchmarking and that data augmentation improves LLM capabilities in various domains, including agent-oriented abilities and reasoning skills.

LLM-as-a-Judge has been widely utilized as an evaluation method in various benchmarks and served as supervised rewards in model training. However, despite their excellence in many domains, potential issues are under-explored, undermining their reliability and the scope of their utility. Therefore, we identify 12 key potential biases and propose a new automated bias quantification framework—CALM—which systematically quantifies and analyzes each type of bias in LLM-as-a-Judge by using automated and principle-guided modification. Our experiments cover multiple popular language models, and the results indicate that while advanced models have achieved commendable overall performance, significant biases persist in certain specific tasks. Empirical results suggest that there remains room for improvement in the reliability of LLM-as-a-Judge. Moreover, we also discuss the explicit and implicit influence of these biases and give some suggestions for the reliable application of LLM-as-a-Judge. Our work highlights the need for stakeholders to address these issues and remind users to exercise caution in LLM-as-a-Judge applications.

This paper presents a novel 8-to-12 b, 1-to-20 MSPS Reconfigurable SAR ADC for low power multi-standard systems to avoid multi-ADCs’ integration in SoC designs thus reducing power and area. To enhance power efficiency across varying conversion rates and resolutions, proposed ADC integrates a custom-designed source-degeneration dynamic comparator along with a reconfigurable asynchronous mechanism. Furthermore, to broaden its applicability, the ADC utilizes an optimized window-switching (OWS) scheme to mitigate dynamic and static performance degradation stemming from capacitor array nonlinearity without the need for additional calibration algorithms. The prototype is designed in 40nm CMOS process with 1.2 V supply and occupied an active area of 0.0445mm2. Post-simulation results show that ADC’s 8b 1MSPS mode achieves SNDR of 48.86 dB, SFDR of 64.49 dB and FoMWof 154.5 fJ/conv-step. And 10b 5MSPS achieves SNDR of 60.89 dB, SFDR of 74.23 dB and FoMWof 45.8 fJ/conv-step. 12b 20MSPS achieves SNDR of 70.33 dB, SFDR of 83.59 dB and FoMWof 16.4 fJ/conv-step.

Current text-to-image (T2I) generation models struggle to align spatial composition with the input text, especially in complex scenes. Even layout-based approaches yield suboptimal spatial control, as their generation process is decoupled from layout planning, making it difficult to refine the layout during synthesis. We present CoT-Diff, a framework that brings step-by-step CoT-style reasoning into T2I generation by tightly integrating Multimodal Large Language Model (MLLM)-driven 3D layout planning with the diffusion process. CoT-Diff enables layout-aware reasoning inline within a single diffusion round: at each denoising step, the MLLM evaluates intermediate predictions, dynamically updates the 3D scene layout, and continuously guides the generation process. The updated layout is converted into semantic conditions and depth maps, which are fused into the diffusion model via a condition-aware attention mechanism, enabling precise spatial control and semantic injection. Experiments on 3D Scene benchmarks show that CoT-Diff significantly improves spatial alignment and compositional fidelity, and outperforms the state-of-the-art method by 34.7% in complex scene spatial accuracy, thereby validating the effectiveness of this entangled generation paradigm.

The symmetric Password-Authenticated Key Exchange (PAKE) protocol enables two parties sharing a low-entropy password to establish a high-entropy session key, offering advantages in simplicity and efficiency. This makes it particularly suitable for Internet of Things (IoT) devices with limited computational resources, positioning it as one of the most effective security methods for authentication and key exchange in IoT environments. In this paper, we analyze a recently proposed efficient symmetric PAKE protocol for IoT, identifying its vulnerability to offline dictionary attacks and its failure to meet the claimed security goals. Building upon the analysis of these design flaws, we present an enhanced protocol, demonstrating its security within the random oracle model. The enhanced protocol retains the protocol flow and computational operations of the original protocol to the greatest extent possible, maintains nearly the same computational efficiency, and simultaneously addresses the vulnerabilities that made the original protocol susceptible to offline dictionary attacks.

Given the common scenario where a trained model confronts significant variations in data distributions different from the training data at test time, Test Time Adaptation (TTA) has emerged as a crucial field of study. Traditional methods in TTA have focused on filtering low-entropy samples to improve model performance, primarily through entropy minimization techniques. However, these approaches exhibit limitations as they often overlook the potential classes of high-entropy samples. This oversight can result in an inadequate utilization of available data, particularly under challenging conditions where model adaptability is critical. In contrast to conventional approaches, our work diverges from the sole emphasis on low-entropy samples by leveraging the rich information contained within ambiguous samples. We demonstrate that reliance solely on entropy minimization is detrimental when dealing with ambiguous samples. To address this, we introduce Cliff, a novel framework designed to learn from ambiguous samples effectively. Concretely, Cliff comprises two innovative components: Dynamic Recognition (DR) and Gap Raising Loss (GRL). DR proposes a method for identifying ambiguous samples and dynamically assigning weights to them, enhancing the model’s focus on potentially informative discrepancies. Whereas the proposed GRL, indeed theoretically proven to be beneficial to the model, guides the model in effectively distinguishing among potential classes by emphasizing the differences in their predictive probabilities. Extensive experiments conducted on CIFAR-10-C and CIFAR-100-C datasets demonstrate Cliff’s state-of-the-art performance. Our results show an average accuracy improvement of 20.24% and 21.12% over the direct use of source domain models on target domains, respectively.

Large language models (LLMs) have garnered significant attention due to their impressive natural language processing (NLP) capabilities. Recently, many studies have focused on the tool utilization ability of LLMs. They primarily investigated how LLMs effectively collaborate with given specific tools. However, in scenarios where LLMs serve as intelligent agents, as seen in applications like AutoGPT and MetaGPT, LLMs are expected to engage in intricate decision-making processes that involve deciding whether to employ a tool and selecting the most suitable tool(s) from a collection of available tools to fulfill user requests. Therefore, in this paper, we introduce MetaTool, a benchmark designed to evaluate whether LLMs have tool usage awareness and can correctly choose tools. Specifically, we create a dataset called ToolE within the benchmark. This dataset contains various types of user queries in the form of prompts that trigger LLMs to use tools, including both single-tool and multi-tool scenarios. Subsequently, we set the tasks for both tool usage awareness and tool selection. We define four subtasks from different perspectives in tool selection, including tool selection with similar choices, tool selection in specific scenarios, tool selection with possible reliability issues, and multi-tool selection. We conduct experiments involving eight popular LLMs and find that the majority of them still struggle to effectively select tools, highlighting the existing gaps between LLMs and genuine intelligent agents. However, through the error analysis, we found there is still significant room for improvement. Finally, we conclude with insights for tool developers -- we strongly recommend that tool developers choose an appropriate rewrite model for generating new descriptions based on the downstream LLM the tool will apply to.

Multimodal Large Language Models (MLLMs) have gained significant attention recently, showing remarkable potential in artificial general intelligence. However, assessing the utility of MLLMs presents considerable challenges, primarily due to the absence multimodal benchmarks that align with human preferences. Drawing inspiration from the concept of LLM-as-a-Judge within LLMs, this paper introduces a novel benchmark, termed MLLM-as-a-Judge, to assess the ability of MLLMs in assisting judges across diverse modalities, encompassing three distinct tasks: Scoring Evaluation, Pair Comparison, and Batch Ranking. Our study reveals that, while MLLMs demonstrate remarkable human-like discernment in Pair Comparisons, there is a significant divergence from human preferences in Scoring Evaluation and Batch Ranking tasks. Furthermore, a closer examination reveals persistent challenges in the evaluative capacities of LLMs, including diverse biases, hallucinatory responses, and inconsistencies in judgment, even in advanced models such as GPT-4V. These findings emphasize the pressing need for enhancements and further research efforts to be undertaken before regarding MLLMs as fully reliable evaluators. In light of this, we advocate for additional efforts dedicated to supporting the continuous development within the domain of MLLM functioning as judges. The code and dataset are publicly available at our project homepage: https://mllm-judge.github.io/.

Large language models (LLMs) have gained considerable attention for their excellent natural language processing capabilities. Nonetheless, these LLMs present many challenges, particularly in the realm of trustworthiness. This paper introduces TrustLLM, a comprehensive study of trustworthiness in LLMs, including principles for different dimensions of trustworthiness, established benchmark, evaluation, and analysis of trustworthiness for mainstream LLMs, and discussion of open challenges and future directions. Specifically, we first propose a set of principles for trustworthy LLMs that span eight different dimensions. Based on these principles, we further establish a benchmark across six dimensions including truthfulness, safety, fairness, robustness, privacy, and machine ethics. We then present a study evaluating 16 mainstream LLMs in TrustLLM, consisting of over 30 datasets. Our findings firstly show that in general trustworthiness and capability (i.e., functional effectiveness) are positively related. Secondly, our observations reveal that proprietary LLMs generally outperform most open-source counterparts in terms of trustworthiness, raising concerns about the potential risks of widely accessible open-source LLMs. However, a few open-source LLMs come very close to proprietary ones, suggesting that open-source models can achieve high levels of trustworthiness without additional mechanisms like moderator, offering valuable insights for developers in this field. Thirdly, it is important to note that some LLMs may be overly calibrated towards exhibiting trustworthiness, to the extent that they compromise their utility by mistakenly treating benign prompts as harmful and consequently not responding. Besides these observations, we’ve uncovered key insights into the multifaceted trustworthiness in LLMs. We emphasize the importance of ensuring transparency not only in the models themselves but also in the technologies that underpin trustworthiness. We advocate that the establishment of an AI alliance between industry, academia, the open-source community to foster collaboration is imperative to advance the trustworthiness of LLMs.

Large Language Models (LLMs) have achieved remarkable success across various industries and applications, owing to their exceptional generative capabilities. Nevertheless, honesty and helpfulness, which ensure safe and useful real-world deployments, have been considered as the longstanding cornerstones in practice. In this paper, we first established comprehensive principles for honesty LLM and further created the HoneSet with 930 queries across six categories, which is designed to evaluate LLMs’ ability to maintain honesty. Then, we improved the honesty and helpfulness of LLMs in both training-free and fine-tuning settings. Specifically, we propose a training-free method named Curiosity-Driven Prompting, which enables LLMs to express their internal confusion and uncertainty about the given query and then optimize their responses. Moreover, we also propose a two-stage fine-tuning approach, inspired by curriculum learning, to enhance the honesty and helpfulness of LLMs. The method first teaches LLMs to distinguish between honest and dishonest, and then LLMs are trained to learn to respond more helpfully. Experimental results demonstrated that both of the two proposed methods improve the helpfulness of LLMs while making them maintain honesty. Our research has paved the way for more reliable and trustworthy LLMs in real-world applications.

This paper presents a subthreshold-leakage-current-based fully CMOS temperature-to-digital converter with high accuracy and supply rejection. The subthreshold current ratio is constructed by different channel lengths of the same MOSFET type, providing high accuracy and less corner dependence. In addition, the supply sensitivity is enhanced by the proposed subthreshold-leakage-current-based sensing element (SE) and the frequency ratio of two identical currents to frequency converters (CFCs). The prototype was implemented in a 180nm CMOS process. It achieves an inaccuracy of ±0.8°C ($3\sigma$) from 0°C to 100°C after two-point calibration with a resolution of 120mK. Over a wide supply range from 0.8V to 1.6V, the temperature sensor shows a supply sensitivity of 0.9°/V at 30°C. Over the temperature range of 0–100°C, the power supply sensitivity is smaller than 3.4°C/V. Operating at 1V, the sensor has a power consumption of 20nW at 30°C, leading to an FoM of 14.4 pJ$\cdot \text{K}^{2}$.

This article presents a 12-bit column-parallel two-step single-slope analog-to-digital converter (SS ADC). With the merging of analog memory capacitor and input sampling capacitor, the proposed two-step SS ADC realizes simultaneously residue storage and zero-cross detection. The fixed decision point guarantees a static comparator offset. A constant input common-mode level resistor ramp generator, which exploits a current-mode R-2R digital-to-analog converter (DAC) and a variable feedback R-string DAC, is developed to enhance ADC linearity limited by finite common-mode rejection ratio (CMRR) of the operational amplifier. Using a bottom-up foreground self-calibration, harmonic distortion caused by both parasitic capacitor and resistor mismatch is mitigated. This prototype is fabricated using a 130-nm CMOS process. The proposed two-step SS ADC consumes 62-$\mu \text{W}$power when operating at a 100-KS/s sampling frequency and yields a peak spurious-free dynamic range (SFDR) of 76.47 dB with a signal-to-noise-and-distortion ratio (SNDR) of 60.78 dB. The measured differential nonlinearity (DNL) and the integral nonlinearity (INL) are 0.83/−1 and 4.78/−3.31 LSB, respectively.

This article presents a low-power code-recombination (CR) analog-to-digital converter (ADC) with generic feature extraction for wearable electrocardiogram (ECG) sensors in wireless body sensor network (WBSN) applications. The CR ADC features a search forward procedure (SFP) and a search backward procedure (SBP) to hunt for part of the quantization steps cutting down bitcycle and power consumption. Also, CR ADC outputs a digital stream named$K$data served as a compressed feature. A prototyped chip including a proposed ADC is fabricated in a 0.13-$\mu \text{m}$CMOS process. With a 0.6-V supply voltage and a 10-kS/s sampling rate, the measured signal-to-noise-distortion range (SNDR) and spurious-free dynamic range (SFDR) are 58.34 and 70.2 dB, respectively. The ADC consumes only 40-nW power when input residue is within the prediction range defined by ADC’s resolution and the reference voltage, achieving a figure-of-merit (FoM) of 6.2 fJ/conversion-step. The data$K$occupy 2/5 of the raw data and are conducted to categorize cardiovascular diseases illustrating at least 96% accuracy.

This brief presents a second-order noise-shaping (NS) successive approximation register (SAR) analog-to-digital converter (ADC) with two passive integrators. Due to the separation of the preamplifier, these two integrators become independent of each other and the size of the second integrator can be reduced. The NS SAR also realizes the zeros optimization of the noise transfer function (NTF). The analysis shows the NS performance of the proposed ADC is insensitive to the gain variation of the multipath comparator. To mitigate the harmonic distortion caused by capacitor mismatch, thermometer-code 4-bit MSBs are implemented with data weighted averaging (DWA) technique. The overall architecture is simple and robust, which only requires minor modifications to the standard SAR ADC. A prototype 9-bit NS-SAR ADC is designed and simulated in a 130-nm CMOS process. It consumes 59.9 μW of power when operating at 2-MS/s sampling frequency. The proposed ADC achieves peak Schreier figure of merits (FoMs) of 171.9 dB with 78.69-dB signal-to-noise-and-distortion ratio (SNDR) at an oversampling ratio (OSR) of 8.

This paper presents a novel energy efficient code-recombination analog-to-digital converter (ADC) for low power applications. Dynamic tracking algorithm, search forward procedure (SFP) and search backward procedure (SBP) are introduced in this manuscript. Also, to generate the test voltage sequence fed in comparator, a binary code factor (BCF) is presented. And a lookup table (LUT) for digitizing the output code is employed. To verify the algorithm, a 10-bit ADC is designed in 0.13μm CMOS process with a 0.6 V supply. Given a 41.5 Hz sinusoid signal, the proposed ADC exhibits 9.75 effective number of bit (ENOB) and 80.9dB spur-free dynamic range (SFDR) at 10k Hz sample rate. Given full-scale sinusoid signals whose frequency are under 160Hz, the ADC achieves 39-77.4nW power consumption with 2.19-10.8 bitcycles in average, respectively. Also, simulation result shows the DNL and INL is bounded at 0.117 and 0.245LSBs.

This paper proposes a low-power and area efficient 14-bit Successive Approximation Register (SAR) analog-to-digital converter (ADC) for array sensors. A hybrid capacitor digital-to-analog converter (CDAC), which consist of a 10-bit split CDAC and a 5-bit serial CDAC, is utilized to increase the area efficiency. The total required number of unit capacitors are only 52. A foreground digital calibration is employed to compensate the linearity error caused by the capacitor mismatch and bridge parasitic capacitor. The HSPICE post-layout simulation results show that the peak DNL and INL of the proposed ADC are enhanced from 1.27/-1 LSB and 17.29/-16.24 LSB to 0.74/-0.49 LSB and 1.27/-0.54 LSB, respectively. And ENOB is improved from 9.82bit to 13.65 bit at 48.14-KHz input after calibration. With a power consumption of 59 μW, the FOM is 45.42 fJ/step. The CDAC occupies an active area of 15 × 800 μm2and the area efficiency ADC core is only 0.934 μm2/code.