OBJECTIVE: Evaluate the association between telemedicine intensity and ambulatory physician electronic health record (EHR) use following the COVID-19 pandemic. MATERIALS AND METHODS: This retrospective study included ambulatory physicians in 11 specialties at 2 large academic medical centers (Washington University in St Louis [WashU], University of California San Francisco [UCSF]). EHR use measures, including time-based and frequency-based, were analyzed in the post-COVID-19 period (March 1, 2021, through March 7, 2022). Multivariable regression models with 2-way fixed effects were used to assess the association between telemedicine intensity and EHR use. RESULTS: Fully telemedicine physician-weeks were associated with higher EHR (hours per 8 patient scheduled hours; β = 3.2 at WashU, β = 1.4 at UCSF; P < .001) and documentation time (β = 2.7 at WashU, β = 1.4 at UCSF; P < .001). Several differences in discrete EHR-based tasks were observed: fully telemedicine physician-days were associated with lesser ordering, and there were mixed patterns for information seeking and clinical communication tasks. DISCUSSION: Expanded use of telemedicine was associated with significant changes in physician EHR use post-COVID-19 onset. Increased EHR time may suggest a shift in workload, whereas decreased ordering may suggest constraints in virtual care, such as ability to perform physical examination and the reliance on patient-reported symptoms. Institutional differences usage patterns suggest that telemedicine's impact is context-specific and provides opportunities for understanding how to optimize EHRs to support telemedicine. CONCLUSION: Telemedicine shifts physician EHR. Supporting physicians through optimized EHR tools, tailored workflows, and team-based interventions is essential for sustainable virtual care delivery without exacerbating EHR burden.

OBJECTIVE: Early detection of surgical complications allows for timely therapy and proactive risk mitigation. Machine learning (ML) can be leveraged to identify and predict patient risks for postoperative complications. We developed and validated the effectiveness of predicting postoperative complications using a novel surgical Variational Autoencoder (surgVAE) that uncovers intrinsic patterns via cross-task and cross-cohort presentation learning. MATERIALS AND METHODS: This retrospective cohort study used data from the electronic health records of adult surgical patients over 4 years (2018-2021). Six key postoperative complications for cardiac surgery were assessed: acute kidney injury, atrial fibrillation, cardiac arrest, deep vein thrombosis or pulmonary embolism, blood transfusion, and other intraoperative cardiac events. We compared surgVAE's prediction performance against widely-used ML models and advanced representation learning and generative models under 5-fold cross-validation. RESULTS: 89 246 surgeries (49% male, median [IQR] age: 57 [45-69]) were included, with 6502 in the targeted cardiac surgery cohort (61% male, median [IQR] age: 60 [53-70]). surgVAE demonstrated generally superior performance over existing ML solutions across postoperative complications of cardiac surgery patients, achieving macro-averaged AUPRC of 0.409 and macro-averaged AUROC of 0.831, which were 3.4% and 3.7% higher, respectively, than the best alternative method (by AUPRC scores). Model interpretation using Integrated Gradients highlighted key risk factors based on preoperative variable importance. DISCUSSION AND CONCLUSION: Our advanced representation learning framework surgVAE showed excellent discriminatory performance for predicting postoperative complications and addressing the challenges of data complexity, small cohort sizes, and low-frequency positive events. surgVAE enables data-driven predictions of patient risks and prognosis while enhancing the interpretability of patient risk profiles.

OBJECTIVES: To develop and validate a novel measure, action entropy, for assessing the cognitive effort associated with electronic health record (EHR)-based work activities. MATERIALS AND METHODS: EHR-based audit logs of attending physicians and advanced practice providers (APPs) from four surgical intensive care units in 2019 were included. Neural language models (LMs) were trained and validated separately for attendings' and APPs' action sequences. Action entropy was calculated as the cross-entropy associated with the predicted probability of the next action, based on prior actions. To validate the measure, a matched pairs study was conducted to assess the difference in action entropy during known high cognitive effort scenarios, namely, attention switching between patients and to or from the EHR inbox. RESULTS: Sixty-five clinicians performing 5 904 429 EHR-based audit log actions on 8956 unique patients were included. All attention switching scenarios were associated with a higher action entropy compared to non-switching scenarios (P < .001), except for the from-inbox switching scenario among APPs. The highest difference among attendings was for the from-inbox attention switching: Action entropy was 1.288 (95% CI, 1.256-1.320) standard deviations (SDs) higher for switching compared to non-switching scenarios. For APPs, the highest difference was for the to-inbox switching, where action entropy was 2.354 (95% CI, 2.311-2.397) SDs higher for switching compared to non-switching scenarios. DISCUSSION: We developed a LM-based metric, action entropy, for assessing cognitive burden associated with EHR-based actions. The metric showed discriminant validity and statistical significance when evaluated against known situations of high cognitive effort (ie, attention switching). With additional validation, this metric can potentially be used as a screening tool for assessing behavioral action phenotypes that are associated with higher cognitive burden. CONCLUSION: An LM-based action entropy metric-relying on sequences of EHR actions-offers opportunities for assessing cognitive effort in EHR-based workflows.

INTRODUCTION: Research on how people interact with electronic health records (EHRs) increasingly involves the analysis of metadata on EHR use. These metadata can be recorded unobtrusively and capture EHR use at a scale unattainable through direct observation or self-reports. However, there is substantial variation in how metadata on EHR use are recorded, analyzed and described, limiting understanding, replication, and synthesis across studies. RECOMMENDATIONS: In this perspective, we provide guidance to those working with EHR use metadata by describing 4 common types, how they are recorded, and how they can be aggregated into higher-level measures of EHR use. We also describe guidelines for reporting analyses of EHR use metadata-or measures of EHR use derived from them-to foster clarity, standardization, and reproducibility in this emerging and critical area of research.

Raw audit logs provide a comprehensive record of clinicians' activities on an electronic health record (EHR) and have considerable potential for studying clinician behaviors. However, research using raw audit logs is limited because they lack context for clinical tasks, leading to difficulties in interpretation. We describe a novel unsupervised approach using the comparison and visualization of EHR action embeddings to learn context and structure from raw audit log activities. Using a dataset of 15 767 634 raw audit log actions performed by 88 intern physicians over 6 months of EHR use across inpatient and outpatient settings, we demonstrated that embeddings can be used to learn the situated context for EHR-based work activities, identify discrete clinical workflows, and discern activities typically performed across diverse contexts. Our approach represents an important methodological advance in raw audit log research, facilitating the future development of metrics and predictive models to measure clinician behaviors at the macroscale.

OBJECTIVE: Extracorporeal membrane oxygenation (ECMO) resource allocation tools are currently lacking. We developed machine learning (ML) models for predicting COVID-19 patients at risk of receiving ECMO to guide patient triage and resource allocation. MATERIAL AND METHODS: We included COVID-19 patients admitted to intensive care units for >24 h from March 2020 to October 2021, divided into training and testing development and testing-only holdout cohorts. We developed ECMO deployment timely prediction model ForecastECMO using Gradient Boosting Tree (GBT), with pre-ECMO prediction horizons from 0 to 48 h, compared to PaO2/FiO2 ratio, Sequential Organ Failure Assessment score, PREdiction of Survival on ECMO Therapy score, logistic regression, and 30 pre-selected clinical variables GBT Clinical GBT models, with area under the receiver operator curve (AUROC) and precision recall curve (AUPRC) metrics. RESULTS: ECMO prevalence was 2.89% and 1.73% in development and holdout cohorts. ForecastECMO had the best performance in both cohorts. At the 18-h prediction horizon, a potentially clinically actionable pre-ECMO window, ForecastECMO, had the highest AUROC (0.94 and 0.95) and AUPRC (0.54 and 0.37) in development and holdout cohorts in identifying ECMO patients without data 18 h prior to ECMO. DISCUSSION AND CONCLUSIONS: We developed a multi-horizon model, ForecastECMO, with high performance in identifying patients receiving ECMO at various prediction horizons. This model has potential to be used as early alert tool to guide ECMO resource allocation for COVID-19 patients. Future prospective multicenter validation would provide evidence for generalizability and real-world application of such models to improve patient outcomes.

We present an end-to-end model using streaming physiological time series to predict near-term risk for hypoxemia, a rare, but life-threatening condition known to cause serious patient harm during surgery. Inspired by the fact that a hypoxemia event is defined based on a future sequence of low SpO2 (i.e., blood oxygen saturation) instances, we propose the hybrid inference network (hiNet) that makes hybrid inference on both future low SpO2 instances and hypoxemia outcomes. hiNet integrates 1) a joint sequence autoencoder that simultaneously optimizes a discriminative decoder for label prediction, and 2) two auxiliary decoders trained for data reconstruction and forecast, which seamlessly learn contextual latent representations that capture the transition from present states to future states. All decoders share a memory-based encoder that helps capture the global dynamics of patient measurement. For a large surgical cohort of 72,081 surgeries at a major academic medical center, our model outperforms strong baselines including the model used by the state-of-the-art hypoxemia prediction system. With its capability to make real-time predictions of near-term hypoxemic at clinically acceptable alarm rates, hiNet shows promise in improving clinical decision making and easing burden of perioperative care.

Burnout is a significant public health concern affecting nearly half of the healthcare workforce. This paper presents the first end-to-end deep learning framework for predicting physician burnout based on electronic health record (EHR) activity logs, digital traces of physician work activities that are available in any EHR system. In contrast to prior approaches that exclusively relied on surveys for burnout measurement, our framework directly learns deep representations of physician behaviors from large-scale clinician activity logs to predict burnout. We propose the Hierarchical burnout Prediction based on Activity Logs (HiPAL), featuring a pre-trained time-dependent activity embedding mechanism tailored for activity logs and a hierarchical predictive model, which mirrors the natural hierarchical structure of clinician activity logs and captures physicians' evolving burnout risk at both short-term and long-term levels. To utilize the large amount of unlabeled activity logs, we propose a semi-supervised framework that learns to transfer knowledge extracted from unlabeled clinician activities to the HiPAL-based prediction model. The experiment on over 15 million clinician activity logs collected from the EHR at a large academic medical center demonstrates the advantages of our proposed framework in predictive performance of physician burnout and training efficiency over state-of-the-art approaches.

Given the risks and cost of hospitalization, there has been significant interest in exploiting machine learning models to improve perioperative care. However, due to the high dimensionality and noisiness of perioperative data, it remains a challenge to develop accurate and robust encoding for surgical predictions. Furthermore, it is important for the encoding to be interpretable by perioperative care practitioners to facilitate their decision making process. We proposeclinical variational autoencoder (cVAE), a deep latent variable model that addresses the challenges of surgical applications through two salient features. (1) To overcome performance limitations of traditional VAE, it isprediction-guided with explicit expression of predicted outcome in the latent representation. (2) Itdisentangles the latent space so that it can be interpreted in a clinically meaningful fashion. We apply cVAE to two real-world perioperative datasets to evaluate its efficacy and performance in predicting outcomes that are important to perioperative care, including postoperative complication and surgery duration. To demonstrate the generality and facilitate reproducibility, we also apply cVAE to the open MIMIC-III dataset for predicting ICU duration and mortality. Our results show that the latent representation provided by cVAE leads to superior performance in classification, regression and multi-task predictions. The two features of cVAE are mutually beneficial and eliminate the need of a predictor. We further demonstrate the interpretability of the disentangled representation and its capability to capture intrinsic characteristics of hospitalized patients. While this work is motivated by and evaluated in the context of clinical applications, the proposed approach may be generalized for other fields using high-dimensional and noisy data and valuing interpretable representations.

OBJECTIVE: The Anesthesiology Control Tower (ACT) for operating rooms (ORs) remotely assesses the progress of surgeries and provides real-time perioperative risk alerts, communicating risk mitigation recommendations to bedside clinicians. We aim to identify and map ACT-OR nonroutine events (NREs)-risk-inducing or risk-mitigating workflow deviations-and ascertain ACT's impact on clinical workflow and patient safety. MATERIALS AND METHODS: We used ethnographic methods including shadowing ACT and OR clinicians during 83 surgeries, artifact collection, chart reviews for decision alerts sent to the OR, and 10 clinician interviews. We used hybrid thematic analysis informed by a human-factors systems-oriented approach to assess ACT's role and impact on safety, conducting content analysis to assess NREs. RESULTS: Across 83 cases, 469 risk alerts were triggered, and the ACT sent 280 care recommendations to the OR. 135 NREs were observed. Critical factors facilitating ACT's role in supporting patient safety included providing backup support and offering a fresh-eye perspective on OR decisions. Factors impeding ACT included message timing and ACT and OR clinician cognitive lapses. Suggestions for improvement included tailoring ACT message content (structure, timing, presentation) and incorporating predictive analytics for advanced planning. DISCUSSION: ACT served as a safety net with remote surveillance features and as a learning healthcare system with feedback/auditing features. Supporting strategies include adaptive coordination and harnessing clinician/patient support to improve ACT's sustainability. Study insights inform future intraoperative telemedicine design considerations to mitigate safety risks. CONCLUSION: Incorporating similar remote technology enhancement into routine perioperative care could markedly improve safety and quality for millions of surgical patients.

OBJECTIVE: To systematically synthesize and appraise the evidence on the effectiveness of health information technology (HIT)-based discharge care transition interventions (CTIs) on readmissions and emergency room visits. MATERIALS AND METHODS: We conducted a systematic search on multiple databases (MEDLINE, CINAHL, EMBASE, and CENTRAL) on June 29, 2020, targeting readmissions and emergency room visits. Prospective studies evaluating HIT-based CTIs published as original research articles in English language peer-reviewed journals were eligible for inclusion. Outcomes were pooled for narrative analysis. RESULTS: Eleven studies were included for review. Most studies (n = 6) were non-RCTs. Several studies (n = 9) assessed bridging interventions comprised of at least 1 pre- and 1 post-discharge component. The narrative analysis found improvements in patient experience and perceptions of discharge care. DISCUSSION: Given the statistical and clinical heterogeneity among studies, we could not ascertain the cumulative effect of CTIs on clinical outcomes. Nevertheless, we found gaps in current research and its implications for future work, including the need for a HIT-based care transition model for guiding theory-driven design and evaluation of HIT-based discharge CTIs. CONCLUSIONS: We appraised and aggregated empirical evidence on the cumulative effectiveness of HIT-based interventions to support discharge transitions from hospital to home, and we highlighted the implications for evidence-based practice and informatics research.

Electronic health record audit logs capture a time-sequenced record of clinician activities while using the system. Audit log data therefore facilitate unobtrusive measurement at scale of clinical work activities and workflow as well as derivative, behavioral proxies (eg, teamwork). Given its considerable research potential, studies leveraging these data have burgeoned. As the field has matured, the challenges of using the data to answer significant research questions have come into focus. In this Perspective, we draw on our research experiences and insights from the broader audit log literature to advance audit log research. Specifically, we make 2 complementary recommendations that would facilitate substantial progress toward audit log-based measures that are: (1) transparent and validated, (2) standardized to allow for multisite studies, (3) sensitive to meaningful variability, (4) broader in scope to capture key aspects of clinical work including teamwork and coordination, and (5) linked to patient and clinical outcomes.

OBJECTIVE: We utilized a computerized order entry system-integrated function referred to as "void" to identify erroneous orders (ie, a "void" order). Using voided orders, we aimed to (1) identify the nature and characteristics of medication ordering errors, (2) investigate the risk factors associated with medication ordering errors, and (3) explore potential strategies to mitigate these risk factors. MATERIALS AND METHODS: We collected data on voided orders using clinician interviews and surveys within 24 hours of the voided order and using chart reviews. Interviews were informed by the human factors-based SEIPS (Systems Engineering Initiative for Patient Safety) model to characterize the work systems-based risk factors contributing to ordering errors; chart reviews were used to establish whether a voided order was a true medication ordering error and ascertain its impact on patient safety. RESULTS: During the 16-month study period (August 25, 2017, to December 31, 2018), 1074 medication orders were voided; 842 voided orders were true medication errors (positive predictive value = 78.3 ± 1.2%). A total of 22% (n = 190) of the medication ordering errors reached the patient, with at least a single administration, without causing patient harm. Interviews were conducted on 355 voided orders (33% response). Errors were not uniquely associated with a single risk factor, but the causal contributors of medication ordering errors were multifactorial, arising from a combination of technological-, cognitive-, environmental-, social-, and organizational-level factors. CONCLUSIONS: The void function offers a practical, standardized method to create a rich database of medication ordering errors. We highlight implications for utilizing the void function for future research, practice and learning opportunities.

Electronic health records (EHR) use is often considered a significant contributor to clinician burnout. Informatics researchers often measure clinical workload using EHR-derived audit logs and use it for quantifying the contribution of EHR use to clinician burnout. However, translating clinician workload measured using EHR-based audit logs into a meaningful burnout metric requires an alignment with the conceptual and theoretical principles of burnout. In this perspective, we describe a systems-oriented conceptual framework to achieve such an alignment and describe the pragmatic realization of this conceptual framework using 3 key dimensions: standardizing the measurement of EHR-based clinical work activities, implementing complementary measurements, and using appropriate instruments to assess burnout and its downstream outcomes. We discuss how careful considerations of such dimensions can help in augmenting EHR-based audit logs to measure factors that contribute to burnout and for meaningfully assessing downstream patient safety outcomes.

OBJECTIVE: Accurate estimations of surgical case durations can lead to the cost-effective utilization of operating rooms. We developed a novel machine learning approach, using both structured and unstructured features as input, to predict a continuous probability distribution of surgical case durations. MATERIALS AND METHODS: The data set consisted of 53 783 surgical cases performed over 4 years at a tertiary-care pediatric hospital. Features extracted included categorical (American Society of Anesthesiologists [ASA] Physical Status, inpatient status, day of week), continuous (scheduled surgery duration, patient age), and unstructured text (procedure name, surgical diagnosis) variables. A mixture density network (MDN) was trained and compared to multiple tree-based methods and a Bayesian statistical method. A continuous ranked probability score (CRPS), a generalized extension of mean absolute error, was the primary performance measure. Pinball loss (PL) was calculated to assess accuracy at specific quantiles. Performance measures were additionally evaluated on common and rare surgical procedures. Permutation feature importance was measured for the best performing model. RESULTS: MDN had the best performance, with a CRPS of 18.1 minutes, compared to tree-based methods (19.5-22.1 minutes) and the Bayesian method (21.2 minutes). MDN had the best PL at all quantiles, and the best CRPS and PL for both common and rare procedures. Scheduled duration and procedure name were the most important features in the MDN. CONCLUSIONS: Using natural language processing of surgical descriptors, we demonstrated the use of ML approaches to predict the continuous probability distribution of surgical case durations. The more discerning forecast of the ML-based MDN approach affords opportunities for guiding intelligent schedule design and day-of-surgery operational decisions.

Data and information technology are key to every aspect of our response to the current coronavirus disease 2019 (COVID-19) pandemic-including the diagnosis of patients and delivery of care, the development of predictive models of disease spread, and the management of personnel and equipment. The increasing engagement of informaticians at the forefront of these efforts has been a fundamental shift, from an academic to an operational role. However, the past history of informatics as a scientific domain and an area of applied practice provides little guidance or prologue for the incredible challenges that we are now tasked with performing. Building on our recent experiences, we present 4 critical lessons learned that have helped shape our scalable, data-driven response to COVID-19. We describe each of these lessons within the context of specific solutions and strategies we applied in addressing the challenges that we faced.

To reduce the risk of wrong-patient errors, safety experts recommend allowing only one patient chart to be open at a time. Due to the lack of empirical evidence, the number of allowable open charts is often based on anecdotal evidence or institutional preference, and hence varies across institutions. Using an interrupted time series analysis of intercepted wrong-patient medication orders in an emergency department during 2010-2016 (83.6 intercepted wrong-patient events per 100 000 orders), we found no significant decrease in the number of intercepted wrong-patient medication orders during the transition from a maximum of 4 open charts to a maximum of 2 (b = -0.19, P = .33) and no significant increase during the transition from a maximum of 2 open charts to a maximum of 4 (b = 0.08, P = .67). These results have implications regarding decisions about allowable open charts in the emergency department in relation to the impact on workflow and efficiency.

OBJECTIVE: Medication order voiding allows clinicians to indicate that an existing order was placed in error. We explored whether the order voiding function could be used to record and study medication ordering errors. MATERIALS AND METHODS: We examined medication orders from an academic medical center for a 6-year period (2006-2011; n = 5 804 150). We categorized orders based on status (void, not void) and clinician-provided reasons for voiding. We used multivariable logistic regression to investigate the association between order voiding and clinician, patient, and order characteristics. We conducted chart reviews on a random sample of voided orders ( n = 198) to investigate the rate of medication ordering errors among voided orders, and the accuracy of clinician-provided reasons for voiding. RESULTS: We found that 0.49% of all orders were voided. Order voiding was associated with clinician type (physician, pharmacist, nurse, student, other) and order type (inpatient, prescription, home medications by history). An estimated 70 ± 10% of voided orders were due to medication ordering errors. Clinician-provided reasons for voiding were reasonably predictive of the actual cause of error for duplicate orders (72%), but not for other reasons. DISCUSSION AND CONCLUSION: Medication safety initiatives require availability of error data to create repositories for learning and training. The voiding function is available in several electronic health record systems, so order voiding could provide a low-effort mechanism for self-reporting of medication ordering errors. Additional clinician training could help increase the quality of such reporting.

OBJECTIVE: Given the complexities of the healthcare environment, efforts to develop standardized handoff practices have led to widely varying manifestations of handoff tools. A systematic review of the literature on handoff evaluation studies was performed to investigate the nature, methodological, and theoretical foundations underlying the evaluation of handoff tools and their adequacy and appropriateness in achieving standardization goals. METHOD: We searched multiple databases for articles evaluating handoff tools published between 1 February 1983 and 15 June 2012. The selected articles were categorized along the following dimensions: handoff tool characteristics, standardization initiatives, methodological framework, and theoretical perspectives underlying the evaluation. RESULTS: Thirty-six articles met our inclusion criteria. Handoff evaluations were conducted primarily on electronic tools (64%), with a more recent focus on electronic medical record-integrated tools (36% since 2008). Most evaluations centered on intra-departmental tools (95%). Evaluation studies were quasi-experimental (42%) or observational (50%), with a major focus on handoff-related outcome measures (94%) using predominantly survey-based tools (70%) with user satisfaction metrics (53%). Most of the studies (81%) based their evaluation on aspects of standardization that included continuity of care and patient safety. CONCLUSIONS: The nature, methodological, and theoretical foundations of handoff tool evaluations varied significantly in terms of their quality and rigor, thereby limiting their ability to inform strategic standardization initiatives. Future research should utilize rigorous, multi-method qualitative and quantitative approaches that capture the contextual nuances of handoffs, and evaluate their effect on patient-related outcomes.

Our study compared how experts and novices performed exploratory search using a traditional search engine and a social tagging system. As expected, results showed that social tagging systems could facilitate exploratory search for both experts and novices. We, however, also found that experts were better at interpreting the social tags and generating search keywords, which made them better at finding information in both interfaces. Specifically, experts found more general information than novices by better interpretation of social tags in the tagging system; and experts also found more domain-specific information by generating more of their own keywords. We found a dynamic interaction between knowledge-in-the-head and knowledge-in-the-social-web that although information seekers are more and more reliant on information from the social Web, domain expertise is still important in guiding them to find and evaluate the information. Implications on the design of social search systems that facilitate exploratory search are also discussed.

We present an extension of a computational cognitive model of social tagging and exploratory search called the semantic imitation model. The model assumes a probabilistic representation of semantics for both internal and external knowledge, and utilizes social tags as navigational cues during exploratory search. We used the model to generate a measure of information scent that controls exploratory search behavior, and simulated the effects of multiple presentations of navigational cues on both simple information retrieval and exploratory search performance based on a previous model called SNIF-ACT. We found that search performance can be significantly improved by these model-based presentations of navigational cues for both experts and novices. The result suggested that exploratory search performance depends critically on the match between internal knowledge (domain expertise) and external knowledge structures (folksonomies). Results have significant implications on how social information systems should be designed to facilitate knowledge exchange among users with different background knowledge.

We present a semantic imitation model of social tagging and exploratory search based on theories of cognitive science. The model assumes that social tags evoke a spontaneous tag-based topic inference process that primes the semantic interpretation of resource contents during exploratory search, and the semantic priming of existing tags in turn influences future tag choices. The model predicts that (1) users who can see tags created by others tend to create tags that are semantically similar to these existing tags, demonstrating the social influence of tag choices; and (2) users who have similar information goals tend to create tags that are semantically similar, but this effect is mediated by the semantic representation and interpretation of social tags. Results from the experiment comparing tagging behavior between a social group (where participants can see tags created by others) and a nominal group (where participants cannot see tags created by others) confirmed these predictions. The current results highlight the critical role of human semantic representations and interpretation processes in the analysis of large-scale social information systems. The model implies that analysis at both the individual and social levels are important for understanding the active, dynamic processes between human knowledge structures and external folksonomies. Implications on how social tagging systems can facilitate exploratory search, interactive information retrievals, knowledge exchange, and other higher-level cognitive and learning activities are discussed.

Previous research has shown that older adults performed worse in web search tasks, and attributed poorer performance to a decline in their cognitive abilities. We conducted a study involving younger and older adults to compare their web search behavior and performance in ill-defined and well-defined information tasks using a health information website. In ill-defined tasks, only a general description about information needs was given, while in well-defined tasks, information needs as well as the specific target information were given. We found that older adults performed worse than younger adults in well-defined tasks, but the reverse was true in ill-defined tasks. Older adults compensated for their lower cognitive abilities by adopting a top-down knowledge-driven strategy to achieve the same level of performance in the ill-defined tasks. Indeed, path models showed that cognitive abilities, health literacy, and knowledge influenced search strategies adopted by older and younger adults. Design implications are also discussed.

We report results from interviews with HCI design researchers on their perceptions of how their research relates to the more traditional scientific goal of providing explanations. Theories of explanation are prominent in the physical and natural sciences, psychology, the social sciences, and engineering. Little work though has so-far addressed the special case of how results from reflective design of interactive systems can help provide explanations. We found conceptions of explanation in design research to be broader and more inclusive than those commonly found in the philosophy of science. We synthesized concepts from the interviews into a framework which may help researchers understand how their contributions relate to both classical and emergent conceptions of explanation.

To improve existing social bookmarking systems and to design new ones, researchers and practitioners need to understand how to evaluate tagging behavior. In this paper, we analyze over two years of data from CiteULike, a social bookmarking system for tagging academic papers. We propose six tag metrics-tag growth, tag reuse, tag non-obviousness, tag discrimination, tag frequency, and tag patterns-to understand the characteristics of a social bookmarking system. Using these metrics, we suggest possible design heuristics to implement a social bookmarking system for CiteSeer, a popular online scholarly digital library for computer science. We believe that these metrics and design heuristics can be applied to social bookmarking systems in other domains.

Abstract Since spring of 2002 we have been working on a methodology, decision model, and cognitive support system to aid with effective allocation of anti‐terrorism (AT) resources at Marine Corps installations. The work has so far been focused on the military domain, but the model and the software tools developed to implement it are generalizable to a range of commercial and public‐sector settings including industrial parks, corporate campuses, and civic facilities. The approach suggests that anti‐terrorism decision makers determine mitigation project allocations using measures of facility priority and mitigation project utility as inputs to the allocation algorithm. The three‐part hybrid resource allocation model presented here uses multi‐criteria decision‐making techniques to assess facility (e.g., building, hangar) priorities, a utility function to calculate anti‐terrorism project mitigation values (e.g., protective glazing, wall coatings, and stand‐off barriers) and optimization techniques to determine resource allocations across multiple, competing AT mitigation projects. The model has been realized in a cognitive support system developed as a set of loosely coupled Web services. The approach, model, and cognitive support system have been evaluated using the cognitive walkthrough method with prospective system users in the field. In this paper we describe the domain, the problem space, the decision model, the cognitive support system and summary results of early model and system evaluations.