The convergence of large language models (LLMs) and virtual reality (VR) technologies has led to significant breakthroughs across multiple domains, particularly in healthcare and medicine. Owing to its immersive and interactive capabilities, VR technology has demonstrated exceptional utility in surgical simulation, rehabilitation, physical therapy, mental health, and psychological treatment. By creating highly realistic and precisely controlled environments, VR not only enhances the efficiency of medical training but also enables personalized therapeutic approaches for patients. The convergence of LLMs and VR extends the potential of both technologies. LLM-empowered VR can transform medical education through interactive learning platforms and address complex healthcare challenges using comprehensive solutions. This convergence enhances the quality of training, decision-making, and patient engagement, paving the way for innovative healthcare delivery. This study aims to comprehensively review the current applications, research advancements, and challenges associated with these two technologies in healthcare and medicine. The rapid evolution of these technologies is driving the healthcare industry toward greater intelligence and precision, establishing them as critical forces in the transformation of modern medicine.

OBJECTIVES: To provide balanced consideration of the opportunities and challenges associated with integrating Large Language Models (LLMs) throughout the medical school continuum. PROCESS: Narrative review of published literature contextualized by current reports of LLM application in medical education. CONCLUSIONS: LLMs like OpenAI's ChatGPT can potentially revolutionize traditional teaching methodologies. LLMs offer several potential advantages to students, including direct access to vast information, facilitation of personalized learning experiences, and enhancement of clinical skills development. For faculty and instructors, LLMs can facilitate innovative approaches to teaching complex medical concepts and fostering student engagement. Notable challenges of LLMs integration include the risk of fostering academic misconduct, inadvertent overreliance on AI, potential dilution of critical thinking skills, concerns regarding the accuracy and reliability of LLM-generated content, and the possible implications on teaching staff.

OBJECTIVE: To develop a deep learning algorithm (DLA) to detect diabetic kideny disease (DKD) from retinal photographs of patients with diabetes, and evaluate performance in multiethnic populations. MATERIALS AND METHODS: We trained 3 models: (1) image-only; (2) risk factor (RF)-only multivariable logistic regression (LR) model adjusted for age, sex, ethnicity, diabetes duration, HbA1c, systolic blood pressure; (3) hybrid multivariable LR model combining RF data and standardized z-scores from image-only model. Data from Singapore Integrated Diabetic Retinopathy Program (SiDRP) were used to develop (6066 participants with diabetes, primary-care-based) and internally validate (5-fold cross-validation) the models. External testing on 2 independent datasets: (1) Singapore Epidemiology of Eye Diseases (SEED) study (1885 participants with diabetes, population-based); (2) Singapore Macroangiopathy and Microvascular Reactivity in Type 2 Diabetes (SMART2D) (439 participants with diabetes, cross-sectional) in Singapore. Supplementary external testing on 2 Caucasian cohorts: (3) Australian Eye and Heart Study (AHES) (460 participants with diabetes, cross-sectional) and (4) Northern Ireland Cohort for the Longitudinal Study of Ageing (NICOLA) (265 participants with diabetes, cross-sectional). RESULTS: In SiDRP validation, area under the curve (AUC) was 0.826(95% CI 0.818-0.833) for image-only, 0.847(0.840-0.854) for RF-only, and 0.866(0.859-0.872) for hybrid. Estimates with SEED were 0.764(0.743-0.785) for image-only, 0.802(0.783-0.822) for RF-only, and 0.828(0.810-0.846) for hybrid. In SMART2D, AUC was 0.726(0.686-0.765) for image-only, 0.701(0.660-0.741) in RF-only, 0.761(0.724-0.797) for hybrid. DISCUSSION AND CONCLUSION: There is potential for DLA using retinal images as a screening adjunct for DKD among individuals with diabetes. This can value-add to existing DLA systems which diagnose diabetic retinopathy from retinal images, facilitating primary screening for DKD.

OBJECTIVE: The potential of using retinal images as a biomarker of cardiovascular disease (CVD) risk has gained significant attention, but regulatory approval of such artificial intelligence (AI) algorithms is lacking. In this regulated pivotal trial, we validated the efficacy of Reti-CVD, an AI-Software as a Medical Device (AI-SaMD), that utilizes retinal images to stratify CVD risk. MATERIALS AND METHODS: In this retrospective study, we used data from the Cardiovascular and Metabolic Diseases Etiology Research Center-High Risk (CMERC-HI) Cohort. Cox proportional hazard model was used to estimate hazard ratio (HR) trend across the 3-tier CVD risk groups (low-, moderate-, and high-risk) according to Reti-CVD in prediction of CVD events. The cardiac computed tomography-measured coronary artery calcium (CAC), carotid intima-media thickness (CIMT), and brachial-ankle pulse wave velocity (baPWV) were compared to Reti-CVD. RESULTS: A total of 1106 participants were included, with 33 (3.0%) participants experiencing CVD events over 5 years; the Reti-CVD-defined risk groups (low, moderate, and high) were significantly associated with increased CVD risk (HR trend, 2.02; 95% CI, 1.26-3.24). When all variables of Reti-CVD, CAC, CIMT, baPWV, and other traditional risk factors were incorporated into one Cox model, the Reti-CVD risk groups were only significantly associated with increased CVD risk (HR = 2.40 [0.82-7.03] in moderate risk and HR = 3.56 [1.34-9.51] in high risk using low-risk as a reference). DISCUSSION: This regulated pivotal study validated an AI-SaMD, retinal image-based, personalized CVD risk scoring system (Reti-CVD). CONCLUSION: These results led the Korean regulatory body to authorize Reti-CVD.

Although deep learning for Diabetic Retinopathy (DR) screening has shown great success in achieving clinically acceptable accuracy for referable versus non-referable DR, there remains a need to provide more fine-grained grading of the DR severity level as well as automated segmentation of lesions (if any) in the retina images. We observe that the DR severity level of an image is dependent on the presence of different types of lesions and their prevalence. In this work, we adopt a multi-task learning approach to perform the DR grading and lesion segmentation tasks. In light of the lack of lesion segmentation mask ground-truths, we further propose a semi-supervised learning process to obtain the segmentation masks for the various datasets. Experiments results on publicly available datasets and a real world dataset obtained from population screening demonstrate the effectiveness of the multi-task solution over state-of-the-art networks.

Though deep learning systems have achieved high accuracy in detecting diseases from medical images, few such systems have been deployed in highly automated disease screening settings due to lack of trust in how well these systems can generalize to out-of-datasets. We propose to use uncertainty estimates of the deep learning system’s prediction to know when to accept or to disregard its prediction. We evaluate the effectiveness of using such estimates in a real-life application for the screening of diabetic retinopathy. We also generate visual explanation of the deep learning system to convey the pixels in the image that influences its decision. Together, these reveal the deep learning system’s competency and limits to the human, and in turn the human can know when to trust the deep learning system.

Ischemic stroke is a leading cause of death and long-term disability that is difficult to predict reliably. Retinal fundus photography has been proposed for stroke risk assessment, due to its non-invasiveness and the similarity between retinal and cerebral microcirculations, with past studies claiming a correlation between venular caliber and stroke risk. However, it may be that other retinal features are more appropriate. In this paper, extensive experiments with deep learning on six retinal datasets are described. Feature isolation involving segmented vascular tree images is applied to establish the effectiveness of vessel caliber and shape alone for stroke classification, and dataset ablation is applied to investigate model generalizability on unseen sources. The results suggest that vessel caliber and shape could be indicative of ischemic stroke, and sourcespecific features could influence model performance.

This article reviews the current state of automatic classification methodologies to identify Diabetic Macular Edema (DME) versus normal subjects based on Spectral Domain OCT (SD-OCT) data. Addressing this classification problem has valuable interest since early detection and treatment of DME play a major role to prevent eye adverse effects such as blindness. The main contribution of this article is to cover the lack of a public dataset and benchmark suited for classifying DME and normal SD-OCT volumes, providing our own implementation of the most relevant methodologies in the literature. Subsequently, 6 different methods were implemented and evaluated using this common benchmark and dataset to produce reliable comparison.

Cataracts are the leading cause of visual impairment and blindness worldwide. Cataract grading, i.e. assessing the presence and severity of cataracts, is essential for diagnosis and progression monitoring. We present in this work an automatic method for predicting cataract grades from slit-lamp lens images. Different from existing techniques which normally formulate cataract grading as a regression problem, we solve it through reconstruction-based classification, which has been shown to yield higher performance when the available training data is densely distributed within the feature space. To heighten the effectiveness of this reconstruction-based approach, we introduce a new semantic feature representation that facilitates alignment of test and reference images, and include locality constraints on the linear reconstruction to reduce the influence of less relevant reference samples. In experiments on the large ACHIKO-NC database comprised of 5378 images, our system outperforms the state-of-the-art regression methods over a range of evaluation metrics. These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Retinal vascular landmark points such as branching points and crossovers are important features for automatic retinal image matching and vascular abnormality detection. These landmark points can enable automatic screening of large dataset through the detection of vascular network abnormalities (i.e., arteriovenous nicking, retinal vein occlusion) which are important for hypertension and cardiovascular disease prediction. Existing methods for crossover point detection use only local information at each image pixel without considering vascular features to detect crossover positions. This leads to the misclassification of very acute crossovers which are represented by two bifurcation points in the skeleton image. In this article, we propose a robust method that utilizes both local information and vascular geometrical features at the crossing to distinguish crossover from non-crossover points in a retinal image. The proposed method was validated on fifteen high resolution retinal images and the results show that our method achieves higher accuracy than any existing methods. In particular, the proposed method can discover more than 74% (recall) of crossovers with a detection accuracy (fraction of detected crossover points that are correct) of 83% (precision). The detected crossovers provide essential results for the automatic detection of vascular network abnormalities, such as arteriovenous nicking, neovascularization, and retinal vein occlusion.

BACKGROUND: Computer-aided diagnosis for screening utilizes computer-based analytical methodologies to process patient information. Glaucoma is the leading irreversible cause of blindness. Due to the lack of an effective and standard screening practice, more than 50% of the cases are undiagnosed, which prevents the early treatment of the disease. OBJECTIVE: To design an automatic glaucoma diagnosis architecture automatic glaucoma diagnosis through medical imaging informatics (AGLAIA-MII) that combines patient personal data, medical retinal fundus image, and patient's genome information for screening. MATERIALS AND METHODS: 2258 cases from a population study were used to evaluate the screening software. These cases were attributed with patient personal data, retinal images and quality controlled genome data. Utilizing the multiple kernel learning-based classifier, AGLAIA-MII, combined patient personal data, major image features, and important genome single nucleotide polymorphism (SNP) features. RESULTS AND DISCUSSION: Receiver operating characteristic curves were plotted to compare AGLAIA-MII's performance with classifiers using patient personal data, images, and genome SNP separately. AGLAIA-MII was able to achieve an area under curve value of 0.866, better than 0.551, 0.722 and 0.810 by the individual personal data, image and genome information components, respectively. AGLAIA-MII also demonstrated a substantial improvement over the current glaucoma screening approach based on intraocular pressure. CONCLUSIONS: AGLAIA-MII demonstrates for the first time the capability of integrating patients' personal data, medical retinal image and genome information for automatic glaucoma diagnosis and screening in a large dataset from a population study. It paves the way for a holistic approach for automatic objective glaucoma diagnosis and screening.

Glaucoma is a chronic eye disease that leads to vision loss. As it cannot be cured, detecting the disease in time is important. Current tests using intraocular pressure (IOP) are not sensitive enough for population based glaucoma screening. Optic nerve head assessment in retinal fundus images is both more promising and superior. This paper proposes optic disc and optic cup segmentation using superpixel classification for glaucoma screening. In optic disc segmentation, histograms, and center surround statistics are used to classify each superpixel as disc or non-disc. A self-assessment reliability score is computed to evaluate the quality of the automated optic disc segmentation. For optic cup segmentation, in addition to the histograms and center surround statistics, the location information is also included into the feature space to boost the performance. The proposed segmentation methods have been evaluated in a database of 650 images with optic disc and optic cup boundaries manually marked by trained professionals. Experimental results show an average overlapping error of 9.5% and 24.1% in optic disc and optic cup segmentation, respectively. The results also show an increase in overlapping error as the reliability score is reduced, which justifies the effectiveness of the self-assessment. The segmented optic disc and optic cup are then used to compute the cup to disc ratio for glaucoma screening. Our proposed method achieves areas under curve of 0.800 and 0.822 in two data sets, which is higher than other methods. The methods can be used for segmentation and glaucoma screening. The self-assessment will be used as an indicator of cases with large errors and enhance the clinical deployment of the automatic segmentation and screening.

The vertical Cup-to-Disc Ratio (CDR) is an important indicator in the diagnosis of glaucoma. Automatic segmentation of the optic disc (OD) and optic cup is crucial towards a good computer-aided diagnosis (CAD) system. This paper presents a statistical model-based method for the segmentation of the optic disc and optic cup from digital color fundus images. The method combines knowledge-based Circular Hough Transform and a novel optimal channel selection for segmentation of the OD. Moreover, we extended the method to optic cup segmentation, which is a more challenging task. The system was tested on a dataset of 325 images. The average Dice coefficient for the disc and cup segmentation is 0.92 and 0.81 respectively, which improves significantly over existing methods. The proposed method has a mean absolute CDR error of 0.10, which outperforms existing methods. The results are promising and thus demonstrate a good potential for this method to be used in a mass screening CAD system.

Age-related macular degeneration (AMD) is a leading cause of vision loss. The presence of drusen are often associated to AMD. Drusen are tiny yellowish-white extracellular buildup present around the macular region of the retina. Clinically, ophthalmologists examine the area around the macula to determine the presence and severity of drusen. However, manual identification and recognition of drusen is subjective, time consuming and expensive. To reduce manual workload and facilitate large-scale early AMD screening, it is essential to detect drusen automatically. In this paper, we propose to use biologically inspired features (BIF) for the purpose of AMD detection. The optic disc and macula are detected to determine a focal region around macula for feature extraction. The extracted features are then classified using support vector machines (SVM). Our experimental results, tested on 350 images, demonstrate that the biologically inspired features from the focal region is effective for drusen detection with a sensitivity of 86.3% and specificity of 91.9%. The results of our proposed approach can be used to reduce workload of ophthalmologists and diagnosis cost.

Peripapillary atrophy (PPA) is an atrophy of pre-existing retina tissue. Because of its association with eye diseases such as myopia and glaucoma, PPA is an important indicator for diagnosis of these diseases. Experienced ophthalmologists are able to determine the presence of PPA using visual information from the retinal images. However, it is tedious, time consuming and subjective to examine all images especially in a screening program. This paper presents biologically inspired feature (BIF) for the automatic detection of PPA. BIF mimics the process of cortex for visual perception. In the proposed method, a focal region is segmented from the retinal image and the BIF is extracted. As BIF is an intrinsically low dimensional feature embedded in a high dimensional space, it is not suitable to measure the similarity between two BIFs directly based on the Euclidean distance. Therefore, it is necessary to obtain a suitable mapping to reduce the dimensionality. In this paper, we explore sparse transfer learning to transfer the label information from ophthalmologists to the sample distribution knowledge contained in all samples. Selective pair-wise discriminant analysis is used to define two strategies of sparse transfer learning: negative and positive sparse transfer learning. Experimental results show that negative sparse transfer learning is superior to the positive one for this task. The proposed BIF based approach achieves an accuracy of more than 90% in detecting PPA, much better than previous methods. It can be used to save the workload of ophthalmologists and thus reduce the diagnosis costs.

Cataract is a leading cause of blindness worldwide. Computer-aided cataract detection is two-fold significant. Firstly, it will be helpful in mass screening. Secondly, it can be used as the preprocessing step for computer-aided grading. In this paper, the enhanced texture feature is proposed based on the graders' expertise of cataract and the characteristics of the retro-illumination lens images. The statistics of the enhanced texture feature is used to train the linear discriminant analysis to detect the cataract. The accuracy of 84.8% is achieved on a clinical database that contains 4545 pairs of images. It demonstrates that the proposed method is promising for mass screening and as the preprocessing step for computer-aided grading.

We have developed a robust method to perform retinal vascular fractal analysis from digital retina images. The technique preprocesses the green channel retina images with Gabor wavelet transforms to enhance the retinal images. Fourier Fractal dimension is computed on these preprocessed images and does not require any segmentation of the vessels. This novel technique requires human input only at a single step; the allocation of the optic disk center. We have tested this technique on 380 retina images from healthy individuals aged 50+ years, randomly selected from the Blue Mountains Eye Study population. To assess its reliability in assessing retinal vascular fractals from different allocation of optic center, we performed pair-wise Pearson correlation between the fractal dimension estimates with 100 simulated region of interest for each of the 380 images. There was Gaussian distribution variation in the optic center allocation in each simulation. The resulting mean correlation coefficient (standard deviation) was 0.93 (0.005). The repeatability of this method was found to be better than the earlier box-counting method. Using this method to assess retinal vascular fractals, we have also confirmed a reduction in the retinal vasculature complexity with aging, consistent with observations from other human organ systems.

In clinical diagnosis, a grade indicating the severity of nuclear cataract is often manually assigned by a trained ophthalmologist to a patient after comparing the lens' opacity severity in his/her slit-lamp images with a set of standard photos. This grading scheme is often subjective and time-consuming. In this paper, a novel computer-aided diagnosis method via ranking is proposed to facilitate nuclear cataract grading following conventional clinical decision-making process. The grade of nuclear cataract in a slit-lamp image is predicted using its neighboring labeled images in a ranked image list, which is achieved using a learned ranking function. This ranking function is learned via direct optimization on a newly proposed approximation to a ranking evaluation measure. Our proposed method has been evaluated by a large dataset composed of 1000 different cases, which are collected from an ongoing clinical population-based study. Both experimental results and comparison with several existing methods demonstrate the benefit of grading via ranking by our proposed method.

This paper presents a photometric restoration technique that automatically corrects shading within retinal images taken with a fundus camera. The proposed technique is based on the observation that the background of retinal images usually shows flat reflectance variations due to its high similarity in color and texture. It estimates shading through an iterative polynomial interpolation procedure that first estimates a shading image through a horizontal interpolation process and then improves the shading estimation by a vertical interpolation process. Once the shading image is estimated, a reflectance image can accordingly be determined based on the luminance of the retina image under study. Experiments on 161 retinal images of different qualities show promising results.

Computer aided analysis of medical images, a unique type of non-text media, can facilitate clinical diagnosis. As an example, an automatic opacity detection approach is proposed in this paper to grade cortical cataract more objectively. The automatic pupil detection is performed by detecting the strongest edges on the convex hull and ellipse fitting using nonlinear least square method. The cortical opacity is detected by radial edge detection and post-processing. The automatic grades are assigned following Wisconsin cataract grading protocol. The accuracy of pupil detection is 98.2%. The mean error of opacity area detection is 7 percent compared with the result of human grader. And 86.3% accurate grades of cortical cataract are achieved. This is the first time that the spoke-like feature is utilized in the automatic detection of cortical cataract to separate from other opacity types. The encouraging results show that it is probable to apply the proposed approach to clinical diagnosis later.

The temporal registration of retinal images provides an important groundwork for doctors to monitor the progression of diseases. Retinal image registration is challenging because the intensity of the retina and the vascular structure can vary greatly over time. In this paper, we describe a tree matching approach to register retinal images. We model each vessel in a retinal image as a tree, called vessel feature tree (VFT). We design a matching function to compute the similarity of a pair of vessels based on their VFTs. We develop a global alignment algorithm to compute the best match between the vessels in two images. Experiment results on 300 pairs of real-world retina images indicate that the proposed approach is able to achieve an accuracy of 93%