OBJECTIVE: Surgical outcome prediction is challenging but necessary for postoperative management. Current machine learning models utilize pre- and post-op data, excluding intraoperative information in surgical notes. Current models also usually predict binary outcomes even when surgeries have multiple outcomes that require different postoperative management. This study addresses these gaps by incorporating intraoperative information into multimodal models for multiclass glaucoma surgery outcome prediction. MATERIALS AND METHODS: We developed and evaluated multimodal deep learning models for multiclass glaucoma trabeculectomy surgery outcomes using both structured EHR data and free-text operative notes. We compare those to baseline models that use structured EHR data exclusively, or neural network models that leverage only operative notes. RESULTS: The multimodal neural network had the highest performance with a macro AUROC of 0.750 and F1 score of 0.583. It outperformed the baseline machine learning model with structured EHR data alone (macro AUROC of 0.712 and F1 score of 0.486). Additionally, the multimodal model achieved the highest recall (0.692) for hypotony surgical failure, while the surgical success group had the highest precision (0.884) and F1 score (0.775). DISCUSSION: This study shows that operative notes are an important source of predictive information. The multimodal predictive model combining perioperative notes and structured pre- and post-op EHR data outperformed other models. Multiclass surgical outcome prediction can provide valuable insights for clinical decision-making. CONCLUSIONS: Our results show the potential of deep learning models to enhance clinical decision-making for postoperative management. They can be applied to other specialties to improve surgical outcome predictions.

We introduce a new concept of panoramic retinal (panretinal) optical coherence tomography (OCT) imaging system with a 140° field of view (FOV). To achieve this unprecedented FOV, a contact imaging approach was used which enabled faster, more efficient, and quantitative retinal imaging with measurement of axial eye length. The utilization of the handheld panretinal OCT imaging system could allow earlier recognition of peripheral retinal disease and prevent permanent vision loss. In addition, adequate visualization of the peripheral retina has a great potential for better understanding disease mechanisms regarding the periphery. To the best of our knowledge, the panretinal OCT imaging system presented in this manuscript has the widest FOV among all the retina OCT imaging systems and offers significant values in both clinical ophthalmology and basic vision science.

Accurate disease diagnosis requires objective assessment of clinical image quality. Automated image quality assessment (IQA) could enhance screening and diagnosis workflows. However, development of generalizable quality assessment tools requires large labeled clinical image datasets from different sites. Obtaining these datasets is often infeasible; and quality indicators may vary with acquisition settings due to domain shift. We introduce a minimally-supervised image quality assessment (MIQA) approach that can learn effectively with small datasets and limited labels in class-imbalanced domain shift scenarios. We formulate the IQA task as an anomaly detection problem, and use a small number of target domain images to identify a compact subset of source domain data for better representation of acceptable quality features. For this compact source domain dataset, we extract features with a pre-trained CNN, perform adaptive feature selection, and develop a one-class classifier to detect poor quality images. We evaluate our approach on two ophthalmology datasets, and show substantial AUC gains and improved cross-site generalizability over competitive baselines. Our approach has implications for improved image quality audit in many clinical settings.

We study the problem of ranking regression, in which a dataset of rankings is used to learn Plackett–Luce scores as functions of sample features. We propose a novel spectral algorithm to accelerate learning in ranking regression. Our main technical contribution is to show that the Plackett–Luce negative log-likelihood augmented with a proximal penalty has stationary points that satisfy the balance equations of a Markov Chain. This allows us to tackle the ranking regression problem via an efficient spectral algorithm by using the Alternating Directions Method of Multipliers (ADMM). ADMM separates the learning of scores and model parameters, and in turn, enables us to devise fast spectral algorithms for ranking regression via both shallow and deep neural network (DNN) models. For shallow models, our algorithms are up to 579 times faster than the Newton’s method. For DNN models, we extend the standard ADMM via a Kullback–Leibler proximal penalty and show that this is still amenable to fast inference via a spectral approach. Compared to a state-of-the-art siamese network, our resulting algorithms are up to 175 times faster and attain better predictions by up to 26% Top-1 Accuracy and 6% Kendall-Tau correlation over five real-life ranking datasets.

Learning from ranking observations arises in many domains, and siamese deep neural networks have shown excellent inference performance in this setting. However, SGD does not scale well, as an epoch grows exponentially with the ranking observation size. We show that a spectral algorithm can be combined with deep learning methods to significantly accelerate training. We combine a spectral estimate of Plackett-Luce ranking scores with a deep model via the Alternating Directions Method of Multipliers with a Kullback-Leibler proximal penalty. Compared to a state-of-the-art siamese network, our algorithms are up to 175 times faster and attain better predictions by up to 26% Top-1 Accuracy and 6% Kendall-Tau correlation over five real-life ranking datasets.

We consider a ranking regression problem in which we use a dataset of ranked choices to learn Plackett-Luce scores as functions of sample features. We solve the maximum likelihood estimation problem by using the Alternating Directions Method of Multipliers (ADMM), effectively separating the learning of scores and model parameters. This separation allows us to express scores as the stationary distribution of a continuous-time Markov Chain. Using this equivalence, we propose two spectral algorithms for ranking regression that learn model parameters up to 579 times faster than the Newton’s method.

As healthcare providers have transitioned from paper to electronic health records they have gained access to increasingly sophisticated documentation aids such as custom note templates. However, little is known about how providers use these aids. To address this gap, we examine how 48 ophthalmologists and their staff create and use content-importing phrases — a customizable and composable form of note template — to document office visits across two years. In this case study, we find 1) content-importing phrases were used to document the vast majority of visits (95%), 2) most content imported by these phrases was structured data imported by data-links rather than boilerplate text, and 3) providers primarily used phrases they had created while staff largely used phrases created by other people. We conclude by discussing how framing clinical documentation as end-user programming can inform the design of electronic health records and other documentation systems mixing data and narrative text.

OBJECTIVE: To systematically review published literature and identify consistency and variation in the aims, measures, and methods of studies using electronic health record (EHR) audit logs to observe clinical activities. MATERIALS AND METHODS: In July 2019, we searched PubMed for articles using EHR audit logs to study clinical activities. We coded and clustered the aims, measures, and methods of each article into recurring categories. We likewise extracted and summarized the methods used to validate measures derived from audit logs and limitations discussed of using audit logs for research. RESULTS: Eighty-five articles met inclusion criteria. Study aims included examining EHR use, care team dynamics, and clinical workflows. Studies employed 6 key audit log measures: counts of actions captured by audit logs (eg, problem list viewed), counts of higher-level activities imputed by researchers (eg, chart review), activity durations, activity sequences, activity clusters, and EHR user networks. Methods used to preprocess audit logs varied, including how authors filtered extraneous actions, mapped actions to higher-level activities, and interpreted repeated actions or gaps in activity. Nineteen studies validated results (22%), but only 9 (11%) through direct observation, demonstrating varying levels of measure accuracy. DISCUSSION: While originally designed to aid access control, EHR audit logs have been used to observe diverse clinical activities. However, most studies lack sufficient discussion of measure definition, calculation, and validation to support replication, comparison, and cross-study synthesis. CONCLUSION: EHR audit logs have potential to scale observational research but the complexity of audit log measures necessitates greater methodological transparency and validated standards.

In many supervised learning settings, elicited labels comprise pairwise comparisons or rankings of samples. We propose a Bayesian inference model for ranking datasets, allowing us to take a probabilistic approach to ranking inference. Our probabilistic assumptions are motivated by, and consistent with, the so-called Plackett-Luce model. We propose a variational inference method to extract a closed-form Gaussian posterior distribution. We show experimentally that the resulting posterior yields more reliable ranking predictions compared to predictions via point estimates.

Retinopathy of Prematurity (ROP) is a leading cause for childhood blindness worldwide. An automated ROP detection system could significantly improve the chance of a child receiving proper diagnosis and treatment. We propose a means of producing a continuous severity score in an automated fashion, regressed from both (a) diagnostic class labels as well as (b) comparison outcomes. Our generative model combines the two sources, and successfully addresses inherent variability in diagnostic outcomes. In particular, our method exhibits an excellent predictive performance of both diagnostic and comparison outcomes over a broad array of metrics, including AUC, precision, and recall.

Pairwise comparison labels are more informative and less variable than class labels, but generating them poses a challenge: their number grows quadratically in the dataset size. We study a natural experimental design objective, namely, D-optimality, that can be used to identify which K pairwise comparisons to generate. This objective is known to perform well in practice, and is submodular, making the selection approximable via the greedy algorithm. A naïve greedy implementation has O(N2 d2 K) complexity, where N is the dataset size, d is the feature space dimension, and K is the number of generated comparisons. We show that, by exploiting the inherent geometry of the dataset–namely, that it consists of pairwise comparisons–the greedy algorithm's complexity can be reduced to O(N2 (K + d) + N(dK + d2) + d2 K). We apply the same acceleration also to the so-called lazy greedy algorithm. When combined, the above improvements lead to an execution time of less than 1 hour for a dataset with 108 comparisons; the naïve greedy algorithm on the same dataset would require more than 10 days to terminate.

Labels generated by human experts via comparisons exhibit smaller variance compared to traditional sample labels. Collecting comparison labels is challenging over large datasets, as the number of comparisons grows quadratically with the dataset size. We study the following experimental design problem: given a budget of expert comparisons, and a set of existing sample labels, we determine the comparison labels to collect that lead to the highest classification improvement. We study several experimental design objectives motivated by the Bradley-Terry model. The resulting optimization problems amount to maximizing submodular functions. We experimentally evaluate the performance of these methods over synthetic and real-life datasets.

Big data and data science investigations hold great promise for making efficient use of data generated in the course of daily life: from social media transactions, news, and a variety of apps used by a large portion of the world’s population, including data generated for health care and life sciences research. Data science brings new insights when large-scale datasets are brought together to characterize and address complex problems. The past decade has seen a plethora of federal and private investments in biomedical data science collection, organization, and analysis, including the National Institutes of Health’s Big Data to Knowledge program, the Patient-Centered Outcomes Research Institute’s PCORnet, and investments from various industries. The work is maturing and interesting, and exciting results are emerging. Biomedical data science offers new and powerful tools to better understand health and disease through insights gleaned from data. Linking data science advances with knowledge representation and clinical information understanding, which have been traditional topics in the biomedical informatics field since its early days, has the potential to accelerate data-driven discovery. Biomedical informatics has also been addressing data-driven discovery. However, until this decade, examples where big data were available for this type of pursuit were limited. Biomedical informatics has thus evolved and overlaps significantly with biomedical data science, the subfield of data science that is concerned with discoveries using primarily clinical and other health-relevant data. All data science investigations must address important and interesting questions that are relevant to the areas they are applied to, have access to comprehensible datasets, and devise and apply methods robust enough to cope with complex unstructured observations.

Response: Drs Black and Klubert raise 3 points in their letter about the limitations of using electronic health record (EHR) timestamp data for workflow timing and modeling1: (1) Our methodologies may not generalize to other specialties beyond ophthalmology. We agree that different workflows in other medical specialties may not be accurately represented by EHR timestamps. We note, however, that a key part of our methodology involves careful analysis and mapping of those individual workflows to EHR timestamps before using them. We are pleased that our methods appear to work among 27 different providers within a single ophthalmology department, and hope other researchers will be interested in applying them in other settings. (2) Our methodologies do not time the granular details of the patient encounter. We agree that this is correct, but note that the goal of this publication was not to measure specific EHR activities during a patient encounter. Instead, our goals were to estimate clinical exam times and documentation time, and to identify trainee involvement in the encounter. We are currently extending our research to develop methodologies for measuring more granular activities. (3) Our methodologies reflect provider EHR use, which may not match actual provider activity during the patient encounter. We completely agree with this limitation and noted it in the paper. Again, we emphasize that the purpose of our study was studying the use of EHR timestamps in workflow research regardless of whether the EHR activity took place during the encounter or after it. We agree with Drs Klubert and Black’s statement that improvement in EHR user interfaces and clinical workflow is warranted.2–4 Overall, we have found that the large volume of readily available EHR data is a distinct benefit for workflow studies, if used carefully. While imperfect, it can be used for simulation models testing changes to staffing, clinic workflows, clinic resources, etc., since they evaluate relative rather than absolute comparisons. Similarly, EHR timing data can be used to compare the relative difference in clinic times after changes are implemented, assuming that EHR use patterns remained the same. We appreciate Drs Klubert and Black’s anecdotes of their challenges with EHR timestamps in their letter, and encourage them to publish their findings in the future so we can compare our specific results with theirs.

Objective: Outpatient clinics lack guidance for tackling modern efficiency and productivity demands. Workflow studies require large amounts of timing data that are prohibitively expensive to collect through observation or tracking devices. Electronic health records (EHRs) contain a vast amount of timing data - timestamps collected during regular use - that can be mapped to workflow steps. This study validates using EHR timestamp data to predict outpatient ophthalmology clinic workflow timings at Oregon Health and Science University and demonstrates their usefulness in 3 different studies. Materials and Methods: Four outpatient ophthalmology clinics were observed to determine their workflows and to time each workflow step. EHR timestamps were mapped to the workflow steps and validated against the observed timings. Results: The EHR timestamp analysis produced times that were within 3 min of the observed times for >80% of the appointments. EHR use patterns affected the accuracy of using EHR timestamps to predict workflow times. Discussion: EHR timestamps provided a reasonable approximation of workflow and can be used for workflow studies. They can be used to create simulation models, analyze EHR use, and quantify the impact of trainees on workflow. Conclusion: The secondary use of EHR timestamp data is a valuable resource for clinical workflow studies. Sample timestamp data files and algorithms for processing them are provided and can be used as a template for more studies in other clinical specialties and settings.