Large Language Models (LLMs) offer substantial promise for clinical natural language processing (NLP); however, a lack of standardized benchmarking methodologies limits their objective evaluation and practical translation. To address this gap, we introduce ClinBench, an open-source, multi-model, multi-domain benchmarking framework. ClinBench is designed for the rigorous evaluation of LLMs on important structured information extraction tasks (e.g., tumor staging, histologic diagnoses, atrial fibrillation, and social determinants of health) from unstructured clinical notes. The framework standardizes the evaluation pipeline by: (i) operating on consistently structured input datasets; (ii) employing dynamic, YAML-based prompting for uniform task definition; and (iii) enforcing output validation via JSON schemas, supporting robust comparison across diverse LLM architectures. We demonstrate ClinBench through a large-scale study of 11 prominent LLMs (e.g., GPT-4o series, LLaMA3 variants, Mixtral) across three clinical domains using configurations of public datasets (TCGA for lung cancer, MIMIC-IV-ECG for atrial fibrillation, and MIMIC notes for SDOH). Our results reveal significant performance-efficiency trade-offs. For example, when averaged across the four benchmarked clinical extraction tasks, GPT-3.5-turbo achieved a mean F1 score of 0.83 with a mean runtime of 16.8 minutes. In comparison, LLaMA3.1-70b obtained a similar mean F1 of 0.82 but required a substantially longer mean runtime of 42.7 minutes. GPT-4o-mini also presented a favorable balance with a mean F1 of 0.81 and a mean runtime of 13.4 minutes. ClinBench provides a unified, extensible framework and empirical insights for reproducible, fair LLM benchmarking in clinical NLP. By enabling transparent and standardized evaluation, this work advances data-centric AI research, informs model selection based on performance, cost, and clinical priorities, and supports the effective integration of LLMs into healthcare. The framework and evaluation code are publicly available at https://github.com/ismaelvillanuevamiranda/ClinBench/.

OBJECTIVE: Social and behavioral determinants of health (SBDH) are increasingly recognized as essential for prognostication and informing targeted interventions. Clinical notes often contain details about SBDH in unstructured format. Conventional extraction methods for these data tend to be labor intensive, inaccurate, and/or unscalable. In this study, we aim to develop and validate a large language model (LLM)-powered method to extract structured SBDH data from clinical notes through prompt engineering. MATERIALS AND METHODS: We developed SBDH-Reader to extract 6 categories of granular SBDH data by prompting GPT-4o, including employment, housing, marital status, and substance use including alcohol, tobacco, and drug use. SBDH-Reader was developed using 7225 notes from 6382 patients in the MIMIC-III database (2001-2012) and externally validated using 971 notes from 437 patients at The University of Texas Southwestern Medical Center (UTSW; 2022-2023). We evaluated SBDH-Reader's performance against human-annotated ground truths based on precision, recall, F1, and confusion matrix. RESULTS: When tested on the UTSW validation set, SBDH-Reader achieved a macro-average F1 ranging from 0.94 to 0.98 across 6 SBDH categories. For clinically relevant adverse attributes, F1 ranged from 0.96 (employment; housing) to 0.99 (tobacco use). When extracting any adverse attributes across all SBDH categories, SBDH-Reader achieved an F1 of 0.97, recall of 0.97, and precision of 0.98 in the independent validation set. DISCUSSION: SBDH-Reader demonstrated strong performance in extracting structured SBDH data through effective prompt engineering of a general-purpose LLM, without the need for task-specific fine-tuning. Its modular design and adaptability to diverse datasets and documentation patterns support its applicability in real-world clinical settings. CONCLUSION: SBDH-Reader has the potential to serve as a scalable and effective method for collecting real-time, patient-level SBDH data to support clinical research and care.

This paper addresses the nature of data and knowledge, the relation between them, the variety of views as a characteristic of Big Data regarding that data may come from many different sources/views from different viewpoints, and the associated essential issues of data provenance, knowledge provenance, scientific computing integrity, and trust in the data science process. Towards the direction of data-intensive science and engineering, it is of paramount importance to ensure Scientific Computing Integrity (SCI). A failure of SCI may be caused by malicious attacks, natural environmental changes, faults of scientists, operations mistakes, faults of supporting systems, faults of processes, and errors in the data or theories on which a research relies. The complexity of scientific workflows and large provenance graphs as well as various causes for SCI failures make ensuring SCI extremely difficult. Provenance and trust play critical role in evaluating SCI. This paper reports our progress in building a model for provenance-based trust reasoning about SCI.

Digital archives rely on trusted parties, such as certification authorities, to ensure authenticity, integrity and proof of existence protection for documents. In this paper, we analyse the trust assumptions that a verifier has to make in order to trust in the protection of a document. We show that trust fades out in the long term due to the ever-growing number of trusted parties. Despite such a dire prospect, current technologies such as X.509 PKI do not assess trust, thereby leaving verifiers in the dark. We present a certification scheme for documents that provides verifiers with a better assessment of trust than in X.509 PKI. In the proposed scheme, trusted parties are rated based on the correctness of their performance. From the ratings, verifiers can assess quantitatively the trust in the trusted parties for the short term, and in the protection of documents for the long term. The proposed scheme encourages trusted parties to work properly.

Integrated role-based access control (RBAC) and attribute-based access control (ABAC) is emerging as a promising paradigm. This paper proposes a framework that uses attribute-based policies to create a more traditional RBAC model. RBAC has been widely used, but has weaknesses: it is labor-intensive and time-consuming to build a model instance, and a pure RBAC system lacks flexibility to efficiently adapt to changing users, objects, and security policies. Particularly, it is impractical to manually make (and maintain) user to role assignments and role to permission assignments in industrial context characterized by a large number of users and/or security objects. ABAC has features complimentary to RBAC, and merging RBAC and ABAC has become an important research topic. This paper proposes a new approach to integrating ABAC with RBAC, by modeling RBAC in two levels. The aboveground level is a standard RBAC model extended with "environment". This level retains the simplicity of RBAC, supporting RBAC model verification/review. The "underground" level is used to represent security knowledge in terms of attribute-based policies, which automatically create the simple RBAC model in the aboveground level. These attribute-based policies bring to RBAC the advantages of ABAC: they are easy to build and easy to adapt to changes. Using this framework, we tackle the problem of permission assignment for large scale applications. This model is motivated by the characteristics and requirements of industrial control systems, and reflects in part certain approaches and practices common in the industry.

This paper formalizes the semantics of trust and studies the transitivity of trust. On the Web, people and software agents have to interact with "strangers". This makes trust a crucial factor on the Web. Basically trust is established in interaction between two entities and any one entity only has a finite number of direct trust relationships. However, activities on the Web require entities to interact with other unfamiliar or unknown entities. As a promising remedy to this problem, social networks-based trust, in which A trusts B, B trusts C, so A indirectly trusts C, is receiving considerable attention. A necessary condition for trust propagation in social networks is that trust needs to be transitive. However, is trust transitive? What types of trust are transitive and why? There are no theories and models found so far to answer these questions in a formal manner. Most models either directly assume trust transitive or do not give a formal discussion of why trust is transitive. To fill this gap, this paper constructs a logical theory of trust in the form of ontology that gives formal and explicit specification for the semantics of trust. Based on this formal semantics, two types of trust -- trust in belief and trust in performance are identified, the transitivity of trust in belief is formally proved, and the conditions for trust propagation are derived. These results give theoretical evidence to support making trust judgment using social networks on the Web.