The rapid development of Multimodal Large Language Models (MLLMs) has enabled the integration of multiple modalities, including texts and images, within the large language model (LLM) framework. However, texts and images are usually interconnected, forming a multimodal attributed graph (MMAG). It is underexplored how MLLMs can incorporate the relational information (i.e., graph structure) and semantic information (i.e., texts and images) on such graphs for multimodal comprehension and generation. In this paper, we propose GraphGPT-o, which supports omni-multimodal understanding and creation on MMAGs. We first comprehensively study linearization variants to transform semantic and structural information as input for MLLMs. Then, we propose a hierarchical aligner that enables deep graph encoding, bridging the gap between MMAGs and MLLMs. Finally, we explore the inference choices, adapting MLLM to interleaved text and image generation in graph scenarios. Extensive experiments on three datasets from different domains demonstrate the effectiveness of our proposed method. Datasets and codes will be publicly available at https://github.com/YiFang99/GraphGPT-o.

Representation learning on text-attributed graphs (TAGs), where nodes are associated with textual descriptions, is crucial for textual and relational knowledge systems, such as social media and recommendation scenarios. However, state-of-the-art embedding methods for TAGs primarily focus on fine-tuning pre-trained language models (PLMs) using structure-aware training objectives. While effective, these methods are tailored for individual TAG and cannot generalize across various graph scenarios. Given the shared textual space, leveraging multiple TAGs for joint fine-tuning, aligning text and graph structure from different aspects, would be more beneficial. Therefore, we propose the Unified Graph Language Model (UniGLM), a novel foundation model pretrained over multiple TAGs from a variety of domains, which can generalize well to both in-domain and cross-domain graph scenarios. Specifically, UniGLM fine-tunes well-established PLMs (e.g., Sentence-BERT) using a domain-aware contrastive learning objective that unifies structure heterogeneity and node statistics across various domains with an adaptive and learnable positive sample selection scheme. Additionally, a lazy updating module is introduced to speed up training by reducing repetitive encoding of positive samples. Extensive datasets across multiple domains, downstream tasks (node classification and link prediction), and a spectrum of graph backbones (supervised and self-supervised graph models) are conducted to compare UniGLM with state-of-the-art baselines. Our empirical observations suggest that UniGLM can generate informative representations for cross-domain graphs observed in the training. More importantly, UniGLM also exhibits competitive transfer ability in encoding unseen TAGs that are not used for training. This study provides deep insights into how to adapt PLMs to graph data and demonstrates the potential of building foundation model for graph representation learning.

OBJECTIVES: Primary graft dysfunction (PGD) is an essential outcome after the heart transplant, which causes severe complications and symptoms for recipients. The in advance prediction of PGD can help the transplant physician better manage the risks of PGD occurrence for patients. Domain experts have identified some important risk factors leading to PGD. However, a widely accepted PGD prediction method is lacking from a computational perspective. In this work, we focus on the prediction of PGD after heart transplant with machine learning (ML). MATERIALS AND METHODS: With the strong power of artificial intelligence, we propose to design a ML algorithm to precisely predict the PGD with the donor and recipient features. Moreover, we apply the computational method to automatically identify important features and interactions between them. RESULTS: To evaluate the effectiveness of the ML algorithm in PGD prediction, we curated a PGD patients' cohort from the United Network for Organ Sharing database, which contains 8008 recipients. 5 commonly used ML models are used for performance comparison. The multi-layer perceptron model achieves superior performance, as measured by area under the receiver operating characteristic curve (AUROC), at 0.868. We identify the top 20 important features and interactions between donors and recipients. Clinical analyses are conducted on the identified features and interactions. DISCUSSION: We summarize the contributions of this work from three aspects including methodology, clinical analysis, and insights. We discuss the limitations of this work on data, model, and real-world implementation perspectives. Additionally, we further discuss the future directions to extend this work to more organ types and diseases. CONCLUSION: In summary, ML has promising applications in PGD prediction as a computational tool for clinical study. We can also use the ML model to help us identify and discover new risk factors and interactions between donor and recipient.

OBJECTIVES: This study aims to (1) review machine learning (ML)-based models for early infection diagnostic and prognosis prediction in post-acute care (PAC) settings, (2) identify key risk predictors influencing infection-related outcomes, and (3) examine the quality and limitations of these models. MATERIALS AND METHODS: PubMed, Web of Science, Scopus, IEEE Xplore, CINAHL, and ACM digital library were searched in February 2024. Eligible studies leveraged PAC data to develop and evaluate ML models for infection-related risks. Data extraction followed the CHARMS checklist. Quality appraisal followed the PROBAST tool. Data synthesis was guided by the socio-ecological conceptual framework. RESULTS: Thirteen studies were included, mainly focusing on respiratory infections and nursing homes. Most used regression models with structured electronic health record data. Since 2020, there has been a shift toward advanced ML algorithms and multimodal data, biosensors, and clinical notes being significant sources of unstructured data. Despite these advances, there is insufficient evidence to support performance improvements over traditional models. Individual-level risk predictors, like impaired cognition, declined function, and tachycardia, were commonly used, while contextual-level predictors were barely utilized, consequently limiting model fairness. Major sources of bias included lack of external validation, inadequate model calibration, and insufficient consideration of data complexity. DISCUSSION AND CONCLUSION: Despite the growth of advanced modeling approaches in infection-related models in PAC settings, evidence supporting their superiority remains limited. Future research should leverage a socio-ecological lens for predictor selection and model construction, exploring optimal data modalities and ML model usage in PAC, while ensuring rigorous methodologies and fairness considerations.

Imbalanced learning is a fundamental challenge in data mining, where there is a disproportionate ratio of training samples in each class. Over-sampling is an effective technique to tackle imbalanced learning through generating synthetic samples for the minority class. While numerous over-sampling algorithms have been proposed, they heavily rely on heuristics, which could be sub-optimal since we may need different sampling strategies for different datasets and base classifiers, and they cannot directly optimize the performance metric. Motivated by this, we investigate developing a learning-based over-sampling algorithm to optimize the classification performance, which is a challenging task because of the huge and hierarchical decision space. At the high level, we need to decide how many synthetic samples to generate. At the low level, we need to determine where the synthetic samples should be located, which depends on the high-level decision since the optimal locations of the samples may differ for different numbers of samples. To address the challenges, we propose AutoSMOTE, an automated over-sampling algorithm that can jointly optimize different levels of decisions. Motivated by the success of SMOTE and its extensions, we formulate the generation process as a Markov decision process (MDP) consisting of three levels of policies to generate synthetic samples within the SMOTE search space. Then we leverage deep hierarchical reinforcement learning to optimize the performance metric on the validation data. Extensive experiments on six real-world datasets demonstrate that AutoSMOTE significantly outperforms the state-of-the-art resampling algorithms. The code is at https://github.com/daochenzha/autosmote

Action recognition is an important task for video understanding with broad applications. However, developing an effective action recognition solution often requires extensive engineering efforts in building and testing different combinations of the modules and their hyperparameters. In this demo, we present AutoVideo, a Python system for automated video action recognition. AutoVideo is featured for 1) highly modular and extendable infrastructure following the standard pipeline language, 2) an exhaustive list of primitives for pipeline construction, 3) data-driven tuners to save the efforts of pipeline tuning, and 4) easy-to-use Graphical User Interface (GUI). AutoVideo is released under MIT license at https://github.com/datamllab/autovideo

The road accidents are a common cause to the injury and death of people. As reported by The American National Highway Traffic Safety Administration (NHTSA), the drivers drowsiness accounts for nearly 100,000 accidents per year in the United States. Thus, we present a novel drivers drowsiness detection system in this paper using the techniques of deep learning(DL), mobile computing, wearable device and Electroencephalography (EEG). We employ the deep learning architecture designed by ourselves that can be easily implemented on the mobile phone to detect the drowsiness with a high accuracy. The EEG signal we use is only single channel that can be easily obtained by the wearable device. The EEG signal collector is designed and made by ourselves, which is like a hair band that makes the driver easier and more comfortable to wear it. The whole system mainly consists of two parts: one is the hardware consisting of EEG headband and sensor, the other is software consisting of Android application and web platform. The app contains the fine trained model to make real-time prediction based on the EEG signal and alert the driver, while sending the data to the backend synchronously. The web platform provides an interface for the monitor to observe the condition of the driver. Our system achieved an accuracy of 97.09% detecting the drivers drowsiness, which surpasses the SOTA methods. The model's size and predict latency are also within a smaller scale than present models that make it more applicable to mobile and embedded system.