Oral and maxillofacial surgery (OMS) imposes an increasing workload on even the most experienced surgeons due to long operation time, high skill requirements, limited observation field, constrained workspace, and fast-growing patient population. Robot-assisted OMS is particularly challenging, requiring technological advancements to replicate complex surgical workflows executed by human surgeons and novel working concepts to properly address human-machine relationships. We introduced a Surgeon Supervised Autonomous Surgical System (SSASS) aiming to solve emerging bottlenecks in OMS. SSASS custom develops a deep-learning-assisted virtual planning module, a teeth-based monocular camera navigation module, and a six-degree-of-freedom compact robot module to function as surgeons’ auxiliary brain, eye, and hand, respectively. These three modules are further seamlessly integrated to autonomously complete most labor-intensive procedures, while prioritizing surgeons to supervise and be responsible for the overall procedure. Le Fort I experiments on five human head models demonstrated that the surgical results of SSASS closely matched the preoperative plan, with high drilling accuracy and acceptable cutting accuracy under a fundamentally new and significantly simplified surgical workflow. Compared to its existing OMS counterparts, SSASS integrates the latest technologies such as deep learning, medical 3D printing, markerless navigation, virtual reality, and collaborative robotics, providing a comprehensive surgical solution for encompassing the entire OMS loop.

Humans can easily perceive the shapes and textures of grasped objects due to high-density mechanoreceptor networks in the hand. However, replicating this capability in wearable devices with limited sensors remains challenging. Here, we designed a tactile glove equipped with easily accessible sensors, enabling accurate identification of soft objects during grasping. We propose an optimization strategy to eliminate redundant sensors and determine the minimal sensor configuration, which was then integrated into the tactile glove. The results indicate that the minimal sensor configuration (n = 7) attached to the hand achieved accurate identification comparable to that obtained using a larger number of sensors (n = 22) distributed across the hand before elimination. Furthermore, we found that various machine learning classifiers achieved recognition accuracies of up to 90% for soft objects when using the tactile glove. Correlation analyses were conducted to characterize individual contribution and mutual cooperativity of regional tactile forces on the hand during grasping, aiding in the interpretation of sensor selection or elimination in the optimization strategy. Adequate validation and analysis demonstrate that our strategy allows an easy-to-apply solution for identifying soft objects via a tactile glove with a minimal number of sensors, offering valuable insights for guiding the design of tactile sensor layouts in artificial limbs and robotic teleoperation systems.

The inertia of manipulated objects contributes to natural human performance, but its effects on virtual reality (VR) interactions have rarely been investigated. Here, we designed a virtual goal-directed task, in which virtual objects with different masses were moved into a target hole. Based on synchronized kinematic and eye-tracking data, we examined the effects of inertia on participants’ performance during the virtual task in a virtual environment. Our results indicated that hand movements presented greater spatial variability and more discontinuities when the inertial load was removed. It suggested a decline in the ability of motor control and feedback regulation, since the absence of an inertial load weakened the proprioception for sensing limb movements. Eye-movement evidence indicated that increased preferential allocations of visual attention contribute to compensating the weakened proprioceptive cues, supporting the kinematic results. These findings reveal the importance and mechanism of inertial effects on human behaviors in VR interactions.

Radiation therapy plays a crucial role in cancer treatment, necessitating precise delivery of radiation to tumors while sparing healthy tissues over multiple days. Computed tomography (CT) is integral for treatment planning, offering electron density data crucial for accurate dose calculations. However, accurately representing patient anatomy is challenging, especially in adaptive radiotherapy, where CT is not acquired daily. Magnetic resonance imaging (MRI) provides superior soft-tissue contrast. Still, it lacks electron density information, while cone beam CT (CBCT) lacks direct electron density calibration and is mainly used for patient positioning. Adopting MRI-only or CBCT-based adaptive radiotherapy eliminates the need for CT planning but presents challenges. Synthetic CT (sCT) generation techniques aim to address these challenges by using image synthesis to bridge the gap between MRI, CBCT, and CT. The SynthRAD2023 challenge was organized to compare synthetic CT generation methods using multi-center ground truth data from 1080 patients, divided into two tasks: (1) MRI-to-CT and (2) CBCT-to-CT. The evaluation included image similarity and dose-based metrics from proton and photon plans. The challenge attracted significant participation, with 617 registrations and 22/17 valid submissions for tasks 1/2. Top-performing teams achieved high structural similarity indices (≥0.87/0.90) and gamma pass rates for photon (≥98.1%/99.0%) and proton (≥97.3%/97.0%) plans. However, no significant correlation was found between image similarity metrics and dose accuracy, emphasizing the need for dose evaluation when assessing the clinical applicability of sCT. SynthRAD2023 facilitated the investigation and benchmarking of sCT generation techniques, providing insights for developing MRI-only and CBCT-based adaptive radiotherapy. It showcased the growing capacity of deep learning to produce high-quality sCT, reducing reliance on conventional CT for treatment planning.

Self-supervised pretraining attempts to enhance model performance by obtaining effective features from unlabeled data, and has demonstrated its effectiveness in the field of histopathology images. Despite its success, few works concentrate on the extraction of nucleus-level information, which is essential for pathologic analysis. In this work, we propose a novel nucleus-aware self-supervised pretraining framework for histopathology images. The framework aims to capture the nuclear morphology and distribution information through unpaired image-to-image translation between histopathology images and pseudo mask images. The generation process is modulated by both conditional and stochastic style representations, ensuring the reality and diversity of the generated histopathology images for pretraining. Further, an instance segmentation guided strategy is employed to capture instance-level information. The experiments on 7 datasets show that the proposed pretraining method outperforms supervised ones on Kather classification, multiple instance learning, and 5 dense-prediction tasks with the transfer learning protocol, and yields superior results than other self-supervised approaches on 8 semi-supervised tasks. Our project is publicly available at https://github.com/zhiyuns/UNITPathSSL.

Domain Adaptation (DA) has recently been of strong interest in the medical imaging community. While a large variety of DA techniques have been proposed for image segmentation, most of these techniques have been validated either on private datasets or on small publicly available datasets. Moreover, these datasets mostly addressed single-class problems. To tackle these limitations, the Cross-Modality Domain Adaptation (crossMoDA) challenge was organised in conjunction with the 24th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2021). CrossMoDA is the first large and multi-class benchmark for unsupervised cross-modality Domain Adaptation. The goal of the challenge is to segment two key brain structures involved in the follow-up and treatment planning of vestibular schwannoma (VS): the VS and the cochleas. Currently, the diagnosis and surveillance in patients with VS are commonly performed using contrast-enhanced T1 (ceT1) MR imaging. However, there is growing interest in using non-contrast imaging sequences such as high-resolution T2 (hrT2) imaging. For this reason, we established an unsupervised cross-modality segmentation benchmark. The training dataset provides annotated ceT1 scans (N=105) and unpaired non-annotated hrT2 scans (N=105). The aim was to automatically perform unilateral VS and bilateral cochlea segmentation on hrT2 scans as provided in the testing set (N=137). This problem is particularly challenging given the large intensity distribution gap across the modalities and the small volume of the structures. A total of 55 teams from 16 countries submitted predictions to the validation leaderboard. Among them, 16 teams from 9 different countries submitted their algorithm for the evaluation phase. The level of performance reached by the top-performing teams is strikingly high (best median Dice score — VS: 88.4%; Cochleas: 85.7%) and close to full supervision (median Dice score — VS: 92.5%; Cochleas: 87.7%). All top-performing methods made use of an image-to-image translation approach to transform the source-domain images into pseudo-target-domain images. A segmentation network was then trained using these generated images and the manual annotations provided for the source image.

Nuclei instance segmentation on histopathology images is of great clinical value for disease analysis. Generally, fully-supervised algorithms for this task require pixel-wise manual annotations, which is especially time-consuming and laborious for the high nuclei density. To alleviate the annotation burden, we seek to solve the problem through image-level weakly supervised learning, which is underexplored for nuclei instance segmentation. Compared with most existing methods using other weak annotations (scribble, point, etc.) for nuclei instance segmentation, our method is more labor-saving. The obstacle to using image-level annotations in nuclei instance segmentation is the lack of adequate location information, leading to severe nuclei omission or overlaps. In this paper, we propose a novel image-level weakly supervised method, called cyclic learning, to solve this problem. Cyclic learning comprises a front-end classification task and a back-end semi-supervised instance segmentation task to benefit from multi-task learning (MTL). We utilize a deep learning classifier with interpretability as the front-end to convert image-level labels to sets of high-confidence pseudo masks and establish a semi-supervised architecture as the back-end to conduct nuclei instance segmentation under the supervision of these pseudo masks. Most importantly, cyclic learning is designed to circularly share knowledge between the front-end classifier and the back-end semi-supervised part, which allows the whole system to fully extract the underlying information from image-level labels and converge to a better optimum. Experiments on three datasets demonstrate the good generality of our method, which outperforms other image-level weakly supervised methods for nuclei instance segmentation, and achieves comparable performance to fully-supervised methods.

Human motion recognition with high accuracy and fast response speed has long been considered an essential component in human-machine interactive activities such as assistive robotics, medical prosthesis, and wearable electronics. The force myography (FMG) signal has been the focus of much investigation in the search for a reliable and efficient muscular locomotion recognition system. However, the effect of the sensing system on FMG-based locomotion classification accuracy has yet to be understood. This study proposed a novel FMG sensing strategy for human lower limb locomotion classification based on flexible supercapacitive iontronic sensors. Benefiting from the ultrahigh sensitivity (up to 1 nF/mmHg) and low activation pressure (less than 5 mmHg) of the supercapacitive iontronic pressure sensor, FMG signal can be acquired accurately from 5 iontronic sensors strapped to the thigh. In the experiment with 12 subjects, the real-time classification strategy based on sliding window and SVM model gave an average locomotion classification accuracy of 99% for seven categories, including sitting, standing, walking on level ground, ramp ascent, ramp descent, stair ascent, stair descent. Compared with traditional FSR sensors, the result showed that iontronic sensors improved the classification accuracy by up to 10 percentage points in the case of short time window. The implementation of the high sensitivity flexible iontronic sensors in the wearable system brings a valuable tool for detecting small human body pressure signals and has great potential to improve the performance of the human-machine interface in rehabilitation and medical applications.

Potential radioactive hazards in full-dose positron emission tomography (PET) imaging remain a concern, whereas the quality of low-dose images is never desirable for clinical use. So it is of great interest to translate low-dose PET images into full-dose. Previous studies based on deep learning methods usually directly extract hierarchical features for reconstruction. We notice that the importance of each feature is different and they should be weighted dissimilarly so that tiny information can be captured by the neural network. Furthermore, the synthesis on some regions of interest is important in some applications. Here we propose a novel segmentation guided style-based generative adversarial network (SGSGAN) for PET synthesis. (1) We put forward a style-based generator employing style modulation, which specifically controls the hierarchical features in the translation process, to generate images with more realistic textures. (2) We adopt a task-driven strategy that couples a segmentation task with a generative adversarial network (GAN) framework to improve the translation performance. Extensive experiments show the superiority of our overall framework in PET synthesis, especially on those regions of interest.

BACKGROUND: Neoantigen based personalized immune therapies achieve promising results in melanoma and lung cancer, but few neoantigen based models perform well in IDH wild-type GBM, and the association between neoantigen intrinsic features and prognosis remain unclear in IDH wild-type GBM. We presented a novel neoantigen intrinsic feature-based deep learning model (neoDL) to stratify IDH wild-type GBMs into subgroups with different survivals. RESULTS: We first derived intrinsic features for each neoantigen associated with survival, followed by applying neoDL in TCGA data cohort(AUC = 0.988, p value < 0.0001). Leave one out cross validation (LOOCV) in TCGA demonstrated that neoDL successfully classified IDH wild-type GBMs into different prognostic subgroups, which was further validated in an independent data cohort from Asian population. Long-term survival IDH wild-type GBMs identified by neoDL were found characterized by 12 protective neoantigen intrinsic features and enriched in development and cell cycle. CONCLUSIONS: The model can be therapeutically exploited to identify IDH wild-type GBM with good prognosis who will most likely benefit from neoantigen based personalized immunetherapy. Furthermore, the prognostic intrinsic features of the neoantigens inferred from this study can be used for identifying neoantigens with high potentials of immunogenicity.

Lower-limb assisted exoskeletons are widely researched for movement assistance or rehabilitation training. Due to advantages of compliance with human body and lightweight, some cable-driven prototypes have been developed, but most of these can assist only unidirectional movement. In this paper we present an untethered cable-driven ankle exoskeleton that can achieve plantarflexion-dorsiflexion bidirectional motion bilaterally using a pair of single motors. The main weights of the exoskeleton, i.e., the motors, power supplement units, and control units, were placed close to the proximity of the human body, i.e., the waist, to reduce the redundant rotation inertia which would apply on the wearer’s leg. A cable force transmission system based on gear-pulley assemblies was designed to transfer the power from the motor to the end-effector effectively. A cable self-tension device on the power output unit was designed to tension the cable during walking. The gait detection system based on a foot pressure sensor and an inertial measurement unit (IMU) could identify the gait cycle and gait states efficiently. To validate the power output performance of the exoskeleton, a torque tracking experiment was conducted. When the subject was wearing the exoskeleton with power on, the muscle activity of the soleus was reduced by 5.2% compared to the state without wearing the exoskeleton. This preliminarily verifies the positive assistance effect of our exoskeleton. The study in this paper demonstrates the promising application of a lightweight cable-driven exoskeleton on human motion augmentation or rehabilitation.

The visual attributes of cells, such as the nuclear morphology and chromatin openness, are critical for histopathology image analysis. By learning cell-level visual representation, we can obtain a rich mix of features that are highly reusable for various tasks, such as cell-level classification, nuclei segmentation, and cell counting. In this paper, we propose a unified generative adversarial networks architecture with a new formulation of loss to perform robust cell-level visual representation learning in an unsupervised setting. Our model is not only label-free and easily trained but also capable of cell-level unsupervised classification with interpretable visualization, which achieves promising results in the unsupervised classification of bone marrow cellular components. Based on the proposed cell-level visual representation learning, we further develop a pipeline that exploits the varieties of cellular elements to perform histopathology image classification, the advantages of which are demonstrated on bone marrow datasets.

BACKGROUND: Histopathology images are critical for medical diagnosis, e.g., cancer and its treatment. A standard histopathology slice can be easily scanned at a high resolution of, say, 200,000×200,000 pixels. These high resolution images can make most existing imaging processing tools infeasible or less effective when operated on a single machine with limited memory, disk space and computing power. RESULTS: In this paper, we propose an algorithm tackling this new emerging "big data" problem utilizing parallel computing on High-Performance-Computing (HPC) clusters. Experimental results on a large-scale data set (1318 images at a scale of 10 billion pixels each) demonstrate the efficiency and effectiveness of the proposed algorithm for low-latency real-time applications. CONCLUSIONS: The framework proposed an effective and efficient system for extremely large histopathology image analysis. It is based on the multiple instance learning formulation for weakly-supervised learning for image classification, segmentation and clustering. When a max-margin concept is adopted for different clusters, we obtain further improvement in clustering performance.

BACKGROUND: Electronic medical record (EMR) systems have become widely used throughout the world to improve the quality of healthcare and the efficiency of hospital services. A bilingual medical lexicon of Chinese and English is needed to meet the demand for the multi-lingual and multi-national treatment. We make efforts to extract a bilingual lexicon from English and Chinese discharge summaries with a small seed lexicon. The lexical terms can be classified into two categories: single-word terms (SWTs) and multi-word terms (MWTs). For SWTs, we use a label propagation (LP; context-based) method to extract candidates of translation pairs. For MWTs, which are pervasive in the medical domain, we propose a term alignment method, which firstly obtains translation candidates for each component word of a Chinese MWT, and then generates their combinations, from which the system selects a set of plausible translation candidates. RESULTS: We compare our LP method with a baseline method based on simple context-similarity. The LP based method outperforms the baseline with the accuracies: 4.44% Acc1, 24.44% Acc10, and 62.22% Acc100, where AccN means the top N accuracy. The accuracy of the LP method drops to 5.41% Acc10 and 8.11% Acc20 for MWTs. Our experiments show that the method based on term alignment improves the performance for MWTs to 16.22% Acc10 and 27.03% Acc20. CONCLUSIONS: We constructed a framework for building an English-Chinese term dictionary from discharge summaries in the two languages. Our experiments have shown that the LP-based method augmented with the term alignment method will contribute to reduction of manual work required to compile a bilingual sydictionary of clinical terms.

BACKGROUND: Subtypes are widely found in cancer. They are characterized with different behaviors in clinical and molecular profiles, such as survival rates, gene signature and copy number aberrations (CNAs). While cancer is generally believed to have been caused by genetic aberrations, the number of such events is tremendous in the cancer tissue and only a small subset of them may be tumorigenic. On the other hand, gene expression signature of a subtype represents residuals of the subtype-specific cancer mechanisms. Using high-throughput data to link these factors to define subtype boundaries and identify subtype-specific drivers, is a promising yet largely unexplored topic. RESULTS: We report a systematic method to automate the identification of cancer subtypes and candidate drivers. Specifically, we propose an iterative algorithm that alternates between gene expression clustering and gene signature selection. We applied the method to datasets of the pediatric cerebellar tumor medulloblastoma (MB). The subtyping algorithm consistently converges on multiple datasets of medulloblastoma, and the converged signatures and copy number landscapes are also found to be highly reproducible across the datasets. Based on the identified subtypes, we developed a PCA-based approach for subtype-specific identification of cancer drivers. The top-ranked driver candidates are found to be enriched with known pathways in certain subtypes of MB. This might reveal new understandings for these subtypes. CONCLUSIONS: Our study indicates that subtype-signature defines the subtype boundaries, characterizes the subtype-specific processes and can be used to prioritize signature-related drivers.

Cancer is a complex disease and within a cancer, subtypes of patients with distinct behaviors often exist. The subtypes might have been caused by different hits, such as copy number aberrations (CNAs) and point mutations, on different pathways/cells-of-origin in a common tissue/organ. Identifying the subtypes with subtype-specific drivers, i.e., hits, is key to the understanding of cancer and development of novel treatments. Here, we report the development of an integrative method to identify the subtypes of cancer. Specifically, we consider CNAs and their impact on gene expressions. Based on these relations, we propose an iterative approach that alternates between kernel based gene expression clustering and gene signature selection. We applied the method to datasets of the pediatric cancer medulloblastoma (MB). The consensus number of clusters quickly converges to three; and for each of these three subtypes, the signature detection also converges to a consistent set of a few hundred highly functionally related genes. For each of the subtypes, we correlate its signature with the set of within-subtype recurrent CNA-affected genes for identifying drivers. The top-ranked driver candidates are found to be enriched with known pathways in certain subtypes of MB as well as containing novel genes that might reveal new understandings for other subtypes.

System R's bottom-up query optimizer architecture forms the basis of most current commercial database managers. The paper compares the performance of top-down and bottom-up optimizers, using the measure of the number of plans generated during optimization. Top down optimizers are superior according to this measure because they can use upper and lower bounds to avoid generating groups of plans. Early during the optimization of a query, a top-down optimizer can derive upper bounds for the costs of the plans it generates. These bounds are not available to typical bottom-up optimizers since such optimizers generate and cost all subplans before considering larger containing plans. These upper bounds can be combined with lower bounds, based solely on logical properties of groups of logically equivalent subqueries, to eliminate entire groups of plans from consideration. We have implemented such a search strategy, in a top-down optimizer called Columbia. Our performance results show that the use of these bounds is quite effective, while preserving the optimality of the resulting plans. In many circumstances this new search strategy is even more effective than heuristics such as considering only left deep plans.