Large Vision-Language Models (LVLMs) are capable of handling diverse data types such as imaging, text, and physiological signals, and can be applied in various fields. In the medical field, LVLMs have a high potential to offer substantial assistance for diagnosis and treatment. Before that, it is crucial to develop benchmarks to evaluate LVLMs' effectiveness in various medical applications. Current benchmarks are often built upon specific academic literature, mainly focusing on a single domain, and lacking varying perceptual granularities. Thus, they face specific challenges, including limited clinical relevance, incomplete evaluations, and insufficient guidance for interactive LVLMs. To address these limitations, we developed the GMAI-MMBench, the most comprehensive general medical AI benchmark with well-categorized data structure and multi-perceptual granularity to date. It is constructed from 284 datasets across 38 medical image modalities, 18 clinical-related tasks, 18 departments, and 4 perceptual granularities in a Visual Question Answering (VQA) format. Additionally, we implemented a lexical tree structure that allows users to customize evaluation tasks, accommodating various assessment needs and substantially supporting medical AI research and applications. We evaluated 50 LVLMs, and the results show that even the advanced GPT-4o only achieves an accuracy of 53.96\%, indicating significant room for improvement. Moreover, we identified five key insufficiencies in current cutting-edge LVLMs that need to be addressed to advance the development of better medical applications. We believe that GMAI-MMBench will stimulate the community to build the next generation of LVLMs toward GMAI.

Large training datasets almost always contain examples with inaccurate or incorrect labels. Deep Neural Networks (DNNs) tend to overfit training label noise, resulting in poorer model performance in practice. To address this problem, we propose a robust label correction framework combining the ideas of small loss selection and noise correction, which learns network parameters and reassigns ground truth labels iteratively. Taking the expertise of DNNs to learn meaningful patterns before fitting noise, our framework first trains two networks over the current dataset with small loss selection. Based on the classification loss and agreement loss of two networks, we can measure the confidence of training data. More and more confident samples are selected for label correction during the learning process. We demonstrate our method on both synthetic and real-world datasets with different noise types and rates, including CIFAR-10, CIFAR-100 and Clothing1M, where our method outperforms the baseline approaches.

Retinal template matching and registration is an important challenge in teleophthalmology with low-cost imaging devices. However, the images from such devices generally have a small field of view (FOV) and image quality degradations, making matching difficult. In this paper, we develop an efficient and accurate retinal matching technique that combines dimension reduction and mutual information (MI), called RetinaMatch. The dimension reduction initializes the MI optimization as a coarse localization process, which narrows the optimization domain and avoids local optima. The effectiveness of RetinaMatch is demonstrated on the open fundus image database STARE with simulated reduced FOV and anticipated degradations, and on retinal images acquired by adapter-based optics attached to a smartphone. RetinaMatch achieves a success rate over 94% on human retinal images with the matched target registration errors below 2 pixels on average, excluding the observer variability, outperforming standard template matching solutions. In the application of measuring vessel diameter repeatedly, single pixel errors are expected. In addition, our method can be used in the process of image mosaicking with area-based registration, providing a robust approach when feature-based methods fail. To the best of our knowledge, this is the first template matching algorithm for retina images with small template images from unconstrained retinal areas. In the context of the emerging mixed reality market, we envision automated retinal image matching and registration methods as transformative for advanced teleophthalmology and long-term retinal monitoring.

Medical robots have been widely used to assist surgeons to carry out dexterous surgical tasks via various ways. Most of the tasks require surgeon's operation directly or indirectly. Certain level of autonomy in robotic surgery could not only free the surgeon from some tedious repetitive tasks, but also utilize the advantages of robot: high dexterity and accuracy. This paper presents a semi-autonomous neurosurgical procedure of brain tumor ablation using RAVEN Surgical Robot and stereo visual feedback. By integrating with the behavior tree framework, the whole surgical task is modeled flexibly and intelligently as nodes and leaves of a behavior tree. This paper provides three contributions mainly: (1) describing the brain tumor ablation as an ideal candidate for autonomous robotic surgery, (2) modeling and implementing the semi-autonomous surgical task using behavior tree framework, and (3) designing an experimental simulated ablation task for feasibility study and robot performance analysis.

This paper considers the semi-automated robotic surgical procedure for removing the brain tumor margins, where the manual operation is a tedious and time-consuming task for surgeons. We present robust path planning methods for robotic ablation of tumor residues in various shapes, which are represented in point-clouds instead of analytical geometry. Along with the path plans, corresponding metrics are also delivered to the surgeon for selecting the optimal candidate in the automated robotic ablation. The selected path plan is then executed and tested on RAVEN™ II surgical robot platform as part of the semi-automated robotic brain tumor ablation surgery in a simulated tissue phantom.

For the goal of designing a wearable low vision aid, aspects of both head-mounted display (HMD) design and performance evaluation were integrated into a single study of scanning ability. A head-mounted version of a novel retinal light scanning display known as the virtual retinal display was fabricated for this study. A remote head CCD (charge coupled device) attached approximately at the user's line of sight was used as the input source. Scanning ability was quantified as the time to identify a target in a wide field of distractors while using the HMD design in four different display interface modes (DIMs). Each DIM was tested with respect to their corresponding controls: A (augmented, see-through) and CO (center occluded) DIMs were compared to the augmented control (augmented, retinal display turned off), and CPO (center and periphery occluded) and PO (periphery occluded) DIMs were compared to the periphery occluded control (periphery obstructed, retinal display turned off). Each DIM was tested at high, medium, and low contrast levels. Five subjects were tested without optical correction (visual acuity worse than 20/200), which accurately represented low vision subjects. Results showed that for each DIM, scanning performance decreased as the contrast level decreased. At the lowest contrast level, the PO DIM provided the greatest and most significant improvement in scanning performance. At the medium and highest contrast levels, all DIMs provided equal or worse performance than their controls. Additionally, the subjects' natural visual acuity showed no correlation with scan time. All subjects reported a higher acuity with the retinal display, and showed a diverse range of preference for the most helpful DIM. Based on these results, we realized that low vision applications have unique design requirements unlike those for normally sighted users, and therefore customization of a wearable low vision aid may be an optimal design strategy.

The extreme environment of a military cockpit requires a novel display technology, introduced as the virtual retinal display (VRD). A head-worn VRD generates an image by optical scanning light directly to the viewer's eye. This novel display allows the direct coupling of the display to an infrared optical head tracking system, resulting in an interactive VRD. With only minor adjustments, the interactive VRD can be used in many non-military AR and VR line-of-sight target tracking applications, such as image plane manipulations. Since the unique tracking technology shares the same aperture or scanned optical beam with the visual display, the tracking produces high accuracy and computational efficiency without the motion artifacts from video frame rate tracking. The apparatus and performance of static and dynamic 3D target tracking in 2D projection overlay are demonstrated.