Language model outputs are not always reliable, thus prompting research into how to adapt model responses based on uncertainty. Common approaches include: abstention, where models refrain from generating responses when uncertain; and linguistic calibration, where models hedge their statements using uncertainty quantifiers. However, abstention can withhold valuable information, while linguistically calibrated responses are often challenging to leverage in downstream tasks. We propose a unified view, Conformal Linguistic Calibration (CLC), which reinterprets linguistic calibration as answer set prediction. First we present a framework connecting abstention and linguistic calibration through the lens of linguistic pragmatics. We then describe an implementation of CLC that allows for controlling the level of imprecision in model responses. Results demonstrate our method produces calibrated outputs with conformal guarantees on factual accuracy. Further, our approach enables fine-tuning models to perform uncertainty-aware adaptive claim rewriting, offering a controllable balance between factuality and specificity.

Identity-Based Multi-proxy Signature (IBMPS) enables legitimate signature delegation by forming a proxy group when the original signer is unavailable. In drone networks, this signature method holds considerable promise, particularly in scenarios such as emergency rescue and military operations. Drones can communicate directly with the execution office on behalf of ground control stations, effectively reducing intermediaries in the information transmission. Recently, numerous communication schemes based on IBMPS have been proposed. However, they still exhibit certain limitations. These schemes often overlook situations where drones may malfunction or be captured, resulting in their inability to properly participate in the collaboration process. Moreover, most existing schemes fail to adequately address the protection of drone anonymity and the mitigation of security threats such as replay attacks. To tackle these challenges, this paper proposes a novel cluster-based and synergistically-assisted scheme built on IBMPS. The scheme leverages bilinear pairings and effectively reduces the communication and computational burden on leader nodes. It also employs a token-based mechanism to dynamically delegate communication initiation rights to the leader nodes. A reputation mechanism is adopted to exclude malicious drones dynamically. Finally, an analysis will be conducted to evaluate the security, communication, and computational costs of the scheme. The performance of the proposed protocol is evaluated using the OMNeT++ simulator, focusing on metrics such as energy consumption, packet loss rate, and end-to-end delay. Experimental results indicate that this scheme has significant advantages in terms of security, with relatively low computational and time costs, and reasonable communication costs.

Zhengping Jiang, Yining Lu, Hanjie Chen, Daniel Khashabi, Benjamin Van Durme, Anqi Liu. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Nathaniel Weir, Kate Sanders, Orion Weller, Shreya Sharma, Dongwei Jiang, Zhengping Jiang, Bhavana Dalvi Mishra, Oyvind Tafjord, Peter Jansen, Peter Clark, Benjamin Van Durme. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

Abstract Computational linguistics models commonly target the prediction of discrete—categorical—labels. When assessing how well-calibrated these model predictions are, popular evaluation schemes require practitioners to manually determine a binning scheme: grouping labels into bins to approximate true label posterior. The problem is that these metrics are sensitive to binning decisions. We consider two solutions to the binning problem that apply at the stage of data annotation: collecting either distributed (redundant) labels or direct scalar value assignment. In this paper, we show that although both approaches address the binning problem by evaluating instance-level calibration, direct scalar assignment is significantly more cost-effective. We provide theoretical analysis and empirical evidence to support our proposal for dataset creators to adopt scalar annotation protocols to enable a higher-quality assessment of model calibration.

We investigate model calibration in the setting of zero-shot cross-lingual transfer with largescale pre-trained language models.The level of model calibration is an important metric for evaluating the trustworthiness of predictive models.There exists an essential need for model calibration when natural language models are deployed in critical tasks.We study different post-training calibration methods in structured and unstructured prediction tasks.We find that models trained with data from the source language become less calibrated when applied to the target language and that calibration errors increase with intrinsic task difficulty and relative sparsity of training data.Moreover, we observe a potential connection between the level of calibration error and an earlier proposed measure of the distance from English to other languages.Finally, our comparison demonstrates that among other methods Temperature Scaling (TS) generalizes well to distant languages, but TS fails to calibrate more complex confidence estimation in structured predictions compared to more expressive alternatives like Gaussian Process Calibration.

David Wan, Chris Kedzie, Faisal Ladhak, Elsbeth Turcan, Petra Galuscakova, Elena Zotkina, Zhengping Jiang, Peter Bell, Kathleen McKeown. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. 2021.

We introduce Uncertain Natural Language Inference (UNLI), a refinement of Natural Language Inference (NLI) that shifts away from categorical labels, targeting instead the direct prediction of subjective probability assessments.We demonstrate the feasibility of collecting annotations for UNLI by relabeling a portion of the SNLI dataset under a probabilistic scale, where items even with the same categorical label differ in how likely people judge them to be true given a premise.We describe a direct scalar regression modeling approach, and find that existing categorically labeled NLI data can be used in pre-training.Our best models approach human performance, demonstrating models may be capable of more subtle inferences than the categorical bin assignment employed in current NLI tasks.