OBJECTIVE: Healthcare decisions are increasingly made with the assistance of machine learning (ML). ML has been known to have unfairness-inconsistent outcomes across subpopulations. Clinicians interacting with these systems can perpetuate such unfairness by overreliance. Recent work exploring ML suppression-silencing predictions based on auditing the ML-shows promise in mitigating performance issues originating from overreliance. This study aims to evaluate the impact of suppression on collaboration fairness and evaluate ML uncertainty as desiderata to audit the ML. MATERIALS AND METHODS: We used data from the Vanderbilt University Medical Center electronic health record (n = 58 817) and the MIMIC-IV-ED dataset (n = 363 145) to predict likelihood of death or intensive care unit transfer and likelihood of 30-day readmission using gradient-boosted trees and an artificially high-performing oracle model. We derived clinician decisions directly from the dataset and simulated clinician acceptance of ML predictions based on previous empirical work on acceptance of clinical decision support alerts. We measured performance as area under the receiver operating characteristic curve and algorithmic fairness using absolute averaged odds difference. RESULTS: When the ML outperforms humans, suppression outperforms the human alone (P < 8.2 × 10-6) and at least does not degrade fairness. When the human outperforms the ML, the human is either fairer than suppression (P < 8.2 × 10-4) or there is no statistically significant difference in fairness. Incorporating uncertainty quantification into suppression approaches can improve performance. CONCLUSION: Suppression of poor-quality ML predictions through an auditor model shows promise in improving collaborative human-AI performance and fairness.

OBJECTIVES: Research on artificial intelligence (AI)-based clinical decision-support (AI-CDS) systems has returned mixed results. Sometimes providing AI-CDS to a clinician will improve decision-making performance, sometimes it will not, and it is not always clear why. This scoping review seeks to clarify existing evidence by identifying clinician-level and technology design factors that impact the effectiveness of AI-assisted decision-making in medicine. MATERIALS AND METHODS: We searched MEDLINE, Web of Science, and Embase for peer-reviewed papers that studied factors impacting the effectiveness of AI-CDS. We identified the factors studied and their impact on 3 outcomes: clinicians' attitudes toward AI, their decisions (eg, acceptance rate of AI recommendations), and their performance when utilizing AI-CDS. RESULTS: We retrieved 5850 articles and included 45. Four clinician-level and technology design factors were commonly studied. Expert clinicians may benefit less from AI-CDS than nonexperts, with some mixed results. Explainable AI increased clinicians' trust, but could also increase trust in incorrect AI recommendations, potentially harming human-AI collaborative performance. Clinicians' baseline attitudes toward AI predict their acceptance rates of AI recommendations. Of the 3 outcomes of interest, human-AI collaborative performance was most commonly assessed. DISCUSSION AND CONCLUSION: Few factors have been studied for their impact on the effectiveness of AI-CDS. Due to conflicting outcomes between studies, we recommend future work should leverage the concept of "appropriate trust" to facilitate more robust research on AI-CDS, aiming not to increase overall trust in or acceptance of AI but to ensure that clinicians accept AI recommendations only when trust in AI is warranted.

Data visualizations can be effective and inclusive means for helping people understand health-related data. Yet numerous high-quality studies comparing data visualizations have yielded relatively little practical design guidance because of a lack of clarity about what communicators want their audience to accomplish. When conducting rigorous evaluations of communication (eg, applying the ISO 9186 method), describing the process simply as evaluating "comprehension" or "interpretation" of visualizations fails to do justice to the true range of outcomes being studied. We present newly developed taxonomies of outcome measures and tasks that are guiding a large-scale systematic review of the health numbers communication literature. Using these taxonomies allows a designer to determine whether a specific data presentation format or feature supports or inhibits the desired audience cognitions, feelings, or behaviors. We argue that taking a granular, outcomes-based approach to designing and evaluating information visualization research is essential to deriving practical, actionable knowledge from it.

OBJECTIVES: We sought to analyze interactive visualizations and animations of health probability data (such as chances of disease or side effects) that have been studied in head-to-head comparisons with either static graphics or numerical communications. MATERIALS AND METHODS: Secondary analysis of a large systematic review on ways to communicate numbers in health. RESULTS: We group the research to show that 4 types of animated or interactive visualizations have been studied by multiple researchers: those that simulate experience of probabilistic events; those that demonstrate the randomness of those events; those that reduce information overload by directing attention sequentially to different items of information; and those that promote elaborative thinking. Overall, these 4 types of visualizations do not show strong evidence of improving comprehension, risk perception, or health behaviors over static graphics. DISCUSSION: Evidence is not yet strong that interactivity or animation is more effective than static graphics for communicating probabilities in health. We discuss 2 possibilities: that the most effective visualizations haven't been studied, and that the visualizations aren't effective. CONCLUSION: Future studies should rigorously compare participant performance with novel interactive or animated visualizations against their performance with static visualizations. Such evidence would help determine whether health communicators should emphasize novel interactive visualizations or rely on older forms of visual communication, which may be accessible to broader audiences, including those with limited digital access.

The 21st Century Cures Act mandates immediate availability of test results upon request. The Cures Act does not require that patients be informed of results, but many organizations send notifications when results become available. Our medical center implemented 2 sequential policies: immediate notifications for all results, and notifications only to patients who opt in. We used over 2 years of data from Vanderbilt University Medical Center to measure the effect of these policies on rates of patient-before-clinician result review and patient-initiated messaging using interrupted time series analysis. When releasing test results with immediate notification, the proportion of patient-before-clinician review increased 4-fold and the proportion of patients who sent messages rose 3%. After transition to opt-in notifications, patient-before-clinician review decreased 2.4% and patient-initiated messaging decreased 0.4%. Replacing automated notifications with an opt-in policy provides patients flexibility to indicate their preferences but may not substantially alleviate clinicians' messaging workload.

Clinical informatics remains underappreciated among medical students in part due to a lack of integration into undergraduate medical education (UME). New developments in the study and practice of medicine are traditionally introduced via formal integration into undergraduate medical curricula. While this path has certain advantages, curricular changes are slow and may fail to showcase the breadth of clinical informatics activities. Less formal and more flexible approaches can circumvent these drawbacks. Interest groups (IGs), which are organized through the Association of American Medical College Careers in Medicine (CiM) program, exemplify the informal approach. CiM IGs are student-led groups that provide exposure to different specialty options, acting as an adjunct to the traditional medical curriculum. While the primary purpose of these groups is to assist students applying to residency programs, we took a novel approach of using an IG to increase student exposure to an area of medicine that had not yet been formally integrated at our institution. IGs provide unique advantages to formal integration into a curriculum as they can be more easily setup and can quickly respond to student interests. Furthermore, IGs can act synergistically with UME, acting as proving grounds for ideas that can lead to new courses. We believe that the lessons and takeaways from our experience can act as a guide for those interested in starting similar organizations at their own schools.

OBJECTIVES: Complex interventions with multiple components and behavior change strategies are increasingly implemented as a form of clinical decision support (CDS) using native electronic health record functionality. Objectives of this study were, therefore, to (1) identify the proportion of randomized controlled trials with CDS interventions that were complex, (2) describe common gaps in the reporting of complexity in CDS research, and (3) determine the impact of increased complexity on CDS effectiveness. MATERIALS AND METHODS: To assess CDS complexity and identify reporting gaps for characterizing CDS interventions, we used the Preferred Reporting Items for Systematic Reviews and Meta-Analyses reporting tool for complex interventions. We evaluated the effect of increased complexity using random-effects meta-analysis. RESULTS: Most included studies evaluated a complex CDS intervention (76%). No studies described use of analytical frameworks or causal pathways. Two studies discussed use of theory but only one fully described the rationale and put it in context of a behavior change. A small but positive effect (standardized mean difference, 0.147; 95% CI, 0.039-0.255; P < .01) in favor of increasing intervention complexity was observed. DISCUSSION: While most CDS studies should classify interventions as complex, opportunities persist for documenting and providing resources in a manner that would enable CDS interventions to be replicated and adapted. Unless reporting of the design, implementation, and evaluation of CDS interventions improves, only slight benefits can be expected. CONCLUSION: Conceptualizing CDS as complex interventions may help convey the careful attention that is needed to ensure these interventions are contextually and theoretically informed.

Qualitative research, the analysis of nonquantitative and nonquantifiable data through methods such as interviews and observation, is integral to the field of biomedical and health informatics. To demonstrate the integrity and quality of their qualitative research, authors should report important elements of their work. This perspective article offers guidance about reporting components of the research, including theory, the research question, sampling, data collection methods, data analysis, results, and discussion. Addressing these points in the paper assists peer reviewers and readers in assessing the rigor of the work and its contribution to the literature. Clearer and more detailed reporting will ensure that qualitative research will continue to be published in informatics, helping researchers disseminate their understanding of people, organizations, context, and sociotechnical relationships as they relate to biomedical and health data.

Use of artificial intelligence in healthcare, such as machine learning-based predictive algorithms, holds promise for advancing outcomes, but few systems are used in routine clinical practice. Trust has been cited as an important challenge to meaningful use of artificial intelligence in clinical practice. Artificial intelligence systems often involve automating cognitively challenging tasks. Therefore, previous literature on trust in automation may hold important lessons for artificial intelligence applications in healthcare. In this perspective, we argue that informatics should take lessons from literature on trust in automation such that the goal should be to foster appropriate trust in artificial intelligence based on the purpose of the tool, its process for making recommendations, and its performance in the given context. We adapt a conceptual model to support this argument and present recommendations for future work.

The objective is to report on the design and evaluation of the inaugural Women in AMIA Leadership Program. A year-long leadership curriculum was developed. Survey responses were summarized with descriptive statistics and quotes selected. Twenty-four scholars participated in the program. There was a significant increase in perceived achievement of learning objectives after the program (P < .0001). The largest improvement was in leadership confidence and presence in work interactions (modal answer Neutral in presurvey from 21 responses rose to Agree in postsurvey from 24 responses). Most (92% of 13) scholars clarified leadership vision and goals and (83% of 18) would be Very Likely to recommend the program to others. The goals of the program-developing women's leader identity, increasing networks, and accumulating experience for future programs-were achieved. The second leadership program is on its way in the United States and Australia. This study may benefit organizations seeking to develop leadership programs for women in informatics and digital health.

OBJECTIVE: Predictive analytics are potentially powerful tools, but to improve healthcare delivery, they must be carefully integrated into healthcare organizations. Our objective was to identify facilitators, challenges, and recommendations for implementing a novel predictive algorithm which aims to prospectively identify patients with high preventable utilization to proactively involve them in preventative interventions. MATERIALS AND METHODS: In preparation for implementing the predictive algorithm in 3 organizations, we interviewed 3 stakeholder groups: health systems operations (eg, chief medical officers, department chairs), informatics personnel, and potential end users (eg, physicians, nurses, social workers). We applied thematic analysis to derive key themes and categorize them into the dimensions of Sittig and Singh's original sociotechnical model for studying health information technology in complex adaptive healthcare systems. Recruiting and analysis were conducted iteratively until thematic saturation was achieved. RESULTS: Forty-nine interviews were conducted in 3 healthcare organizations. Technical components of the implementation (hardware and software) raised fewer concerns than alignment with sociotechnical factors. Stakeholders wanted decision support based on the algorithm to be clear and actionable and incorporated into current workflows. However, how to make this disease-independent classification tool actionable was perceived as a challenge, and appropriate patient interventions informed by the algorithm appeared likely to require substantial external and institutional resources. Stakeholders also described the criticality of trust, credibility, and interpretability of the predictive algorithm. CONCLUSIONS: Although predictive analytics can classify patients with high accuracy, they cannot advance healthcare processes and outcomes without careful implementation that takes into account the sociotechnical system. Key stakeholders have strong perceptions about facilitators and challenges to shape successful implementation.

BACKGROUND: More than 100 studies document disparities in patient portal use among vulnerable populations. Developing and testing strategies to reduce disparities in use is essential to ensure portals benefit all populations. OBJECTIVE: To systematically review the impact of interventions designed to: (1) increase portal use or predictors of use in vulnerable patient populations, or (2) reduce disparities in use. MATERIALS AND METHODS: A librarian searched Ovid MEDLINE, EMBASE, CINAHL, and Cochrane Reviews for studies published before September 1, 2018. Two reviewers independently selected English-language research articles that evaluated any interventions designed to impact an eligible outcome. One reviewer extracted data and categorized interventions, then another assessed accuracy. Two reviewers independently assessed risk of bias. RESULTS: Out of 18 included studies, 15 (83%) assessed an intervention's impact on portal use, 7 (39%) on predictors of use, and 1 (6%) on disparities in use. Most interventions studied focused on the individual (13 out of 26, 50%), as opposed to facilitating conditions, such as the tool, task, environment, or organization (SEIPS model). Twelve studies (67%) reported a statistically significant increase in portal use or predictors of use, or reduced disparities. Five studies (28%) had high or unclear risk of bias. CONCLUSION: Individually focused interventions have the most evidence for increasing portal use in vulnerable populations. Interventions affecting other system elements (tool, task, environment, organization) have not been sufficiently studied to draw conclusions. Given the well-established evidence for disparities in use and the limited research on effective interventions, research should move beyond identifying disparities to systematically addressing them at multiple levels.

Health informatics studies the use of information technology to improve human health. As informaticists, we seek to reduce the gaps between current healthcare practices and our societal goals for better health and healthcare quality, safety, or cost. It is time to recognize health equity as one of these societal goals-a point underscored by this Journal of the American Medical Informatics Association Special Focus Issue, "Health Informatics and Health Equity: Improving our Reach and Impact." This Special Issue highlights health informatics research that focuses on marginalized and underserved groups, health disparities, and health equity. In particular, this Special Issue intentionally showcases high-quality research and professional experiences that encompass a broad range of subdisciplines, methods, marginalized populations, and approaches to disparities. Building on this variety of submissions and other recent developments, we highlight contents of the Special Issue and offer an assessment of the state of research at the intersection of health informatics and health equity.

Dear Dr Ohno-Machado, As Kasthurirathne and colleagues1 point out in their recent JAMIA paper, it is well established that population health is affected by socioeconomic status and other social determinants of health (SDH). It is therefore reasonable to hypothesize that SDH data should have predictive power for clinical and healthcare utilization outcomes for individual patients, yet the authors found that adding SDH data to clinical data produced no significant improvement in the performance of algorithms predicting the need for social service referrals.1 It is important to be reminded that not all available data have utility for all purposes, and that many plausible hypotheses do not survive rigorous analysis. However, we would like to make sure that the informatics community does not interpret these findings more broadly to indicate that SDH data have no value. We would like to propose several possible explanations for these interesting and surprising findings, each of which might suggest future avenues of exploration. Correlations between predictors: It is known that SDH are correlated with the risk of many clinical conditions, and it seems possible that in this particular study, the SDH variables were strongly correlated with the clinical diagnoses. For example, social determinants (eg socioeconomic status, race, social support, etc.) are well established as strong predictors of cardiovascular disease.2 If, in the current study, the information contained in the SDH was already present in the clinical variables, then adding SDH would not improve model performance. Future work in different domains might show SDH to have more predictive power. For example, with certain congenital conditions, SDH might have little influence on the risk of disease but instead could be related to access to care or quality of life for people with the condition, and thus would contribute additional information to a model containing clinical diagnoses. In addition, it is likely that many of the SDH are correlated with each other. Community-level research, which often deals with collinear predictors, often addresses this problem by collapsing correlated measures into scales (such as the Centers for Disease Control’s Social Vulnerability Index [https://svi.cdc.gov]), which are then utilized as single metrics. Choice of outcome variables: A closely related potential explanation is that the specific social service referrals chosen in the current study were not well predicted by SDH variables because they were already too well predicted by the clinical ones. For example, referrals to dietitians (one of the study’s outcome variables) might be so strongly predicted by diagnosis of diabetes, hypertension, or congestive heart failure that additional data are not helpful. Similarly, referrals to mental health services might be sufficiently strongly predicted by the presence of psychiatric diagnoses. It is possible that SDH data may have predictive power for other sorts of healthcare utilization and health outcomes, just not for these particular social services. Variability in predictors: As the authors suggest in their discussion, it is possible that the patients of the safety net health system had insufficient variability in SDH. For example, if household income did not range much above or below the poverty line in the entire population of interest, this variable would provide little discriminatory power. It is possible that models built from a more socioeconomically diverse population data set might provide different results. Comprehensiveness of the clinical data: The authors had access to unusually comprehensive clinical data through the Indiana Network for Patient Care, a well-established health information exchange organization, which allowed the researchers to leverage data such as emergency and hospital admissions from other organizations. Healthcare organizations with access to only their own local clinical data may find that SDH variables contribute more to predictive models, precisely because the SDH data might serve as proxies for some of the missing clinical data. Ecological inferences: In the absence of individual-level SDH data, the authors used community-level (ZIP code and census tract level) measures as proxies. It is possible that for certain SDH variables, community estimates are either imprecise or biased. For example, if the within-community variance in education is extremely high, then individual educational attainment will not be predicted well by the neighborhood average, creating lack of precision. Alternately, if the patients who seek care at a safety net hospital tend to be less well educated than their close neighbors, then individual educational attainment will be systematically overestimated by the neighborhood average, creating bias.3 Future work might explore whether community-level data have more utility when they describe neighborhood characteristics (such as, in this study, data about local availability of well-lit walkways or grocery stores) rather than being used to infer individual-level characteristics (such as education). A useful family of methodological approaches to account for these ecological relationships is hierarchical or multilevel models, which explicitly account for the nested structure of the data (individuals within neighborhoods, in this case). Choice of predictor variables: In addition to the rich set of social and environmental factors used by Kasthurirathne and colleagues, it is possible that others not available to the researchers might have predictive power, such as social support and social capital, or detailed employment type.4 Inspecting variable importance measures5 in the models might suggest new hypotheses about which types of SDH have the most predictive utility, and whether these point to other SDH data to collect or obtain from other sources. Given the extensive public health literature on the relationship between social determinants and health, it is exciting to see the health informatics community begin to explore mergers of clinical and SDH data sets. We appreciate the contribution of Kasthurirathne et al. to this emerging literature and welcome additional exploration of the potential utility of social determinants of health in clinical care and predictive analytics. Conflict of interest statement. None declared.

Objective: Parents routinely access young children's medical records, but medical societies strongly recommend confidential care during adolescence, and most medical centers restrict parental records access during the teen years. We sought to assess public opinion about adolescent medical privacy. Materials and Methods: The Cornell National Social Survey (CNSS) is an annual nationwide public opinion survey. We added questions about a) whether parents should be able to see their 16-year-old child's medical record, and b) whether teens would avoid discussing sensitive issues (sex, alcohol) with doctors if parents could see the record. Hypothesizing that highlighting the rationale for adolescent privacy would change opinions, we conducted an experiment by randomizing question order. Results: Most respondents (83.0%) believed that an adolescent would be less likely to discuss sensitive issues with doctors with parental medical record access; responses did not differ by question order (P = .29). Most also believed that parents should have access to teens' records, but support for parental access fell from 77% to 69% among those asked the teen withholding question first (P = .01). Conclusions: Although medical societies recommend confidential care for adolescents, public opinion is largely in favor of parental access. A brief "nudge," asking whether parental access might harm adolescent-doctor communication, increased acceptance of adolescent confidentiality, and could be part of a strategy to prepare parents for electronic patient portal policies that medical centers impose at the beginning of adolescence.

Objective: Medical privacy policies, which are clear-cut for adults and young children, become ambiguous during adolescence. Yet medical organizations must establish unambiguous rules about patient and parental access to electronic patient portals. We conducted a national interview study to characterize the diversity in adolescent portal policies across a range of institutions and determine the factors influencing decisions about these policies. Methods: Within a sampling framework that ensured diversity of geography and medical organization type, we used purposive and snowball sampling to identify key informants. Semi-structured interviews were conducted and analyzed with inductive thematic analysis, followed by a member check. Results: We interviewed informants from 25 medical organizations. Policies established different degrees of adolescent access (from none to partial to complete), access ages (from 10 to 18 years), degrees of parental access, and types of information considered sensitive. Federal and state law did not dominate policy decisions. Other factors in the decision process were: technology capabilities; differing patient population needs; resources; community expectations; balance between information access and privacy; balance between promoting autonomy and promoting family shared decision-making; and tension between teen privacy and parental preferences. Some informants believed that clearer standards would simplify policy-making; others worried that standards could restrict high-quality polices. Conclusions: In the absence of universally accepted standards, medical organizations typically undergo an arduous decision-making process to develop teen portal policies, weighing legal, economic, social, clinical, and technological factors. As a result, portal access policies are highly inconsistent across the United States and within individual states.

Health informatics interventions are designed to help people avoid, recover from, or cope with disease and disability, or to improve the quality and safety of healthcare. Unfortunately, they pose a risk of producing intervention-generated inequalities (IGI) by disproportionately benefiting more advantaged people. In this perspective paper, we discuss characteristics of health-related interventions known to produce IGI, explain why health informatics interventions are particularly vulnerable to this phenomenon, and describe safeguards that can be implemented to improve health equity. We provide examples in which health informatics interventions produced inequality because they were more accessible to, heavily used by, adhered to, or effective for those from socioeconomically advantaged groups. We provide a brief outline of precautions that intervention developers and implementers can take to guard against creating or worsening inequality through health informatics. We conclude by discussing evaluation approaches that will ensure that IGIs are recognized and studied.

OBJECTIVE: Event notifications are real-time, electronic, automatic alerts to providers of their patients' health care encounters at other facilities. Our objective was to examine the effects of organizational capability and related social/organizational issues upon users' perceptions of the impact of event notifications on quality, efficiency, and satisfaction. MATERIALS AND METHODS: We surveyed representatives (n = 49) of 10 organizations subscribing to the Bronx Regional Health Information Organization's event notification services about organizational capabilities, notification information quality, perceived usage, perceived impact, and organizational and respondent characteristics. The response rate was 89%. Average item scores were used to create an individual domain summary score. The association between the impact of event notifications and organizational characteristics was modeled using random-intercept logistic regression models. RESULTS: Respondents estimated that organizations followed up on the majority (83%) of event notifications. Supportive organizational policies were associated with the perception that event notifications improved quality of care (odds ratio [OR] = 2.12; 95% CI, = 1.05, 4.45), efficiency (OR = 2.06; 95% CI = 1.00, 4.21), and patient satisfaction (OR = 2.56; 95% CI = 1.13, 5.81). Higher quality of event notification information was also associated with a perceived positive impact on quality of care (OR = 2.84; 95% CI = 1.31, 6.12), efficiency (OR = 3.04; 95% CI = 1.38, 6.69), and patient satisfaction (OR = 2.96; 95% CI = 1.25, 7.03). CONCLUSIONS: Health care organizations with appropriate processes, workflows, and staff may be better positioned to use event notifications. Additionally, information quality remains critical in users' assessments and perceptions.

OBJECTIVE: Increasing the use of generic medications could help control medical costs. However, educational interventions have limited impact on prescriber behavior, and e-prescribing alerts are associated with high override rates and alert fatigue. Our objective was to evaluate the effect of a less intrusive intervention, a redesign of an e-prescribing interface that provides default options intended to "nudge" prescribers towards prescribing generic drugs. METHODS: This retrospective cohort study in an academic ambulatory multispecialty practice assessed the effects of customizing an e-prescribing interface to substitute generic equivalents for brand-name medications during order entry and allow a one-click override to order the brand-name medication. RESULTS: Among drugs with generic equivalents, the proportion of generic drugs prescribed more than doubled after the interface redesign, rising abruptly from 39.7% to 95.9% (a 56.2% increase; 95% confidence interval, 56.0-56.4%; P < .001). Before the redesign, generic drug prescribing rates varied by therapeutic class, with rates as low as 8.6% for genitourinary products and 15.7% for neuromuscular drugs. After the redesign, generic drug prescribing rates for all but four therapeutic classes were above 90%: endocrine drugs, neuromuscular drugs, nutritional products, and miscellaneous products. DISCUSSION: Changing the default option in an e-prescribing interface in an ambulatory care setting was followed by large and sustained increases in the proportion of generic drugs prescribed at the practice. CONCLUSIONS: Default options in health information technology exert a powerful effect on user behavior, an effect that can be leveraged to optimize decision making.

OBJECTIVES: Contemporary electronic health records (EHRs) offer a wide variety of features, creating opportunities to influence healthcare quality in different ways. This study was designed to assess the relationship between physician use of individual EHR functions and healthcare quality. MATERIALS AND METHODS: Sixty-five providers eligible for "meaningful use" were included. Data were abstracted from office visit records during the study timeframe (183 095 visits with 61 977 patients). Three EHR functions were considered potential predictors: acceptance of best practice alerts, use of order sets, and viewing panel-level reports. Eighteen clinical quality measures from the "meaningful use" program were abstracted. RESULTS: Use of condition-specific best-practice alerts and order sets was associated with better scores on clinical quality measures capturing processes in diabetes, cancer screening, tobacco cessation, and pneumonia vaccination. For example, providers above the median in use of tobacco-related alerts had higher performance on tobacco cessation intervention metrics (median 80.6% vs. 66.7%; P < .001), and providers above the median in use of diabetes order sets had higher quality on diabetes low density lipoprotein (LDL) testing (68.2% vs. 59.5%; P == .001). Post hoc examination of the results showed that the positive associations were with measures of healthcare processes (such as rates of LDL testing), whereas there were no positive associations with measures of healthcare outcomes (such as LDL levels). DISCUSSION: Among primary care providers in the ambulatory setting using a single EHR, intensive use of certain EHR functions was associated with increased adherence to recommended care as measured by performance on electronically reported "meaningful use" quality measures. This study is relevant to current policy as it uses quality metrics constructed by contemporary certified EHR technology, and quantitative EHR use metrics rather than self-reported use. CONCLUSION: In the early stages of the "meaningful use" program, use of specific EHR functions was associated with higher performance on healthcare process metrics.

BACKGROUND: Studies of the effects of electronic health records (EHRs) have had mixed findings, which may be attributable to unmeasured confounders such as individual variability in use of EHR features. OBJECTIVE: To capture physician-level variations in use of EHR features, associations with other predictors, and usage intensity over time. METHODS: Retrospective cohort study of primary care providers eligible for meaningful use at a network of federally qualified health centers, using commercial EHR data from January 2010 through June 2013, a period during which the organization was preparing for and in the early stages of meaningful use. RESULTS: Data were analyzed for 112 physicians and nurse practitioners, consisting of 430,803 encounters with 99,649 patients. EHR usage metrics were developed to capture how providers accessed and added to patient data (eg, problem list updates), used clinical decision support (eg, responses to alerts), communicated (eg, printing after-visit summaries), and used panel management options (eg, viewed panel reports). Provider-level variability was high: for example, the annual average proportion of encounters with problem lists updated ranged from 5% to 60% per provider. Some metrics were associated with provider, patient, or encounter characteristics. For example, problem list updates were more likely for new patients than established ones, and alert acceptance was negatively correlated with alert frequency. CONCLUSIONS: Providers using the same EHR developed personalized patterns of use of EHR features. We conclude that physician-level usage of EHR features may be a valuable additional predictor in research on the effects of EHRs on healthcare quality and costs.

BACKGROUND: Providing patients with access to their medical data is widely expected to help educate and empower them to manage their own health. Health information exchange (HIE) infrastructures could potentially help patients access records across multiple healthcare providers. We studied three HIE organizations as they developed portals to give consumers access to HIE data previously exchanged only among healthcare organizations. OBJECTIVE: To follow the development of new consumer portal technologies, and to identify barriers and facilitators to patient access to HIE data. METHODS: Semistructured interviews of 15 key informants over a 2-year period spanning the development and early implementation of three new projects, coded according to a sociotechnical framework. RESULTS: As the organizations tried to develop functionality that fully served the needs of both providers and patients, plans were altered by technical barriers (primarily related to data standardization) and cultural and legal issues surrounding data access. Organizational changes also played an important role in altering project plans. In all three cases, patient access to data was significantly scaled back from initial plans. CONCLUSIONS: This prospective study revealed how sociotechnical factors previously identified as important in health information technology success and failure helped to shape the evolution of three novel consumer informatics projects. Barriers to providing patients with seamless access to their HIE data were multifactorial. Remedies will have to address technical, organizational, cultural, and other factors.

Electronic health records (EHR) are becoming more common because of the federal EHR incentive programme, which is also promoting electronic health information exchange (HIE). To determine whether consumers' attitudes toward EHR and HIE are associated with experience with doctors using EHR, a nationwide random-digit-dial survey was conducted in December 2011. Of 1603 eligible people contacted, 1000 (63%) participated. Most believed EHR and HIE would improve healthcare quality (66% and 79%, respectively). Respondents whose doctor had an EHR were more likely to believe that these technologies would improve quality (for EHR, OR 2.3; for HIE, OR 1.7). However, experience with physicians using EHR was not associated with privacy concerns. Consumers whose physicians use EHR were more likely to believe that EHR and HIE will improve healthcare when compared to others. However, experience with a physician using an EHR had no relationship with privacy concerns.

With the proliferation of relatively mature health information technology (IT) systems with large numbers of users, it becomes increasingly important to evaluate the effect of these systems on the quality and safety of healthcare. Previous research on the effectiveness of health IT has had mixed results, which may be in part attributable to the evaluation frameworks used. The authors propose a model for evaluation, the Triangle Model, developed for designing studies of quality and safety outcomes of health IT. This model identifies structure-level predictors, including characteristics of: (1) the technology itself; (2) the provider using the technology; (3) the organizational setting; and (4) the patient population. In addition, the model outlines process predictors, including (1) usage of the technology, (2) organizational support for and customization of the technology, and (3) organizational policies and procedures about quality and safety. The Triangle Model specifies the variables to be measured, but is flexible enough to accommodate both qualitative and quantitative approaches to capturing them. The authors illustrate this model, which integrates perspectives from both health services research and biomedical informatics, with examples from evaluations of electronic prescribing, but it is also applicable to a variety of types of health IT systems.

Implementing health information technology (IT) at the community level is a national priority to help improve healthcare quality, safety, and efficiency. However, community-based organizations implementing health IT may not have expertise in evaluation. This study describes lessons learned from experience as a multi-institutional academic collaborative established to provide independent evaluation of community-based health IT initiatives. The authors' experience derived from adapting the principles of community-based participatory research to the field of health IT. To assist other researchers, the lessons learned under four themes are presented: (A) the structure of the partnership between academic investigators and the community; (B) communication issues; (C) the relationship between implementation timing and evaluation studies; and (D) study methodology. These lessons represent practical recommendations for researchers interested in pursuing similar collaborations.

The purpose of this review is to organize various published conceptions of health numeracy and to discuss how health numeracy contributes to the productive use of quantitative information for health. We define health numeracy as the individual-level skills needed to understand and use quantitative health information, including basic computation skills, ability to use information in documents and non-text formats such as graphs, and ability to communicate orally. We also identify two other factors affecting whether a consumer can use quantitative health information: design of documents and other information artifacts, and health-care providers' communication skills. We draw upon the distributed cognition perspective to argue that essential ingredients for the productive use of quantitative health information include not only health numeracy but also good provider communication skills, as well as documents and devices that are designed to enhance comprehension and cognition.

This review describes recent experimental and focus group research on graphics as a method of communication about quantitative health risks. Some of the studies discussed in this review assessed effect of graphs on quantitative reasoning, others assessed effects on behavior or behavioral intentions, and still others assessed viewers' likes and dislikes. Graphical features that improve the accuracy of quantitative reasoning appear to differ from the features most likely to alter behavior or intentions. For example, graphs that make part-to-whole relationships available visually may help people attend to the relationship between the numerator (the number of people affected by a hazard) and the denominator (the entire population at risk), whereas graphs that show only the numerator appear to inflate the perceived risk and may induce risk-averse behavior. Viewers often preferred design features such as visual simplicity and familiarity that were not associated with accurate quantitative judgments. Communicators should not assume that all graphics are more intuitive than text; many of the studies found that patients' interpretations of the graphics were dependent upon expertise or instruction. Potentially useful directions for continuing research include interactions with educational level and numeracy and successful ways to communicate uncertainty about risk.