We present SensorLM, a family of sensor-language foundation models that enable wearable sensor data understanding with natural language. Despite its pervasive nature, aligning and interpreting sensor data with language remains challenging due to the lack of paired, richly annotated sensor-text descriptions in uncurated, real-world wearable data. We introduce a hierarchical caption generation pipeline designed to capture statistical, structural, and semantic information from sensor data. This approach enabled the curation of the largest sensor-language dataset to date, comprising over 59.7 million hours of data from more than 103,000 people. Furthermore, SensorLM extends prominent multimodal pretraining architectures (e.g., CLIP, CoCa) and recovers them as specific variants within a generic architecture. Extensive experiments on real-world tasks in human activity analysis and healthcare verify the superior performance of SensorLM over state-of-the-art in zero-shot recognition, few-shot learning, and cross-modal retrieval. SensorLM also demonstrates intriguing capabilities including scaling behaviors, label efficiency, sensor captioning, and zero-shot generalization to unseen tasks. Code is available at https://github.com/Google-Health/consumer-health-research/tree/main/sensorlm.

MOTIVATION: Longitudinal studies increasingly collect rich 'omics' data sampled frequently over time and across large cohorts to capture dynamic health fluctuations and disease transitions. However, the generation of longitudinal omics data has preceded the development of analysis tools that can efficiently extract insights from such data. In particular, there is a need for statistical frameworks that can identify not only which omics features are differentially regulated between groups but also over what time intervals. Additionally, longitudinal omics data may have inconsistencies, including non-uniform sampling intervals, missing data points, subject dropout and differing numbers of samples per subject. RESULTS: In this work, we developed OmicsLonDA, a statistical method that provides robust identification of time intervals of temporal omics biomarkers. OmicsLonDA is based on a semi-parametric approach, in which we use smoothing splines to model longitudinal data and infer significant time intervals of omics features based on an empirical distribution constructed through a permutation procedure. We benchmarked OmicsLonDA on five simulated datasets with diverse temporal patterns, and the method showed specificity greater than 0.99 and sensitivity greater than 0.87. Applying OmicsLonDA to the iPOP cohort revealed temporal patterns of genes, proteins, metabolites and microbes that are differentially regulated in male versus female subjects following a respiratory infection. In addition, we applied OmicsLonDA to a longitudinal multi-omics dataset of pregnant women with and without preeclampsia, and OmicsLonDA identified potential lipid markers that are temporally significantly different between the two groups. AVAILABILITY AND IMPLEMENTATION: We provide an open-source R package (https://bioconductor.org/packages/OmicsLonDA), to enable widespread use. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Diet management is key to managing chronic dis-eases such as diabetes. Automated food recommender systems may be able to assist by providing meal recommendations that conform to a user’s nutrition goals and food preferences. Current recommendation systems suffer from a lack of accuracy that is in part due to a lack of knowledge of food preferences. In this work, we propose a method for learning food preferences from food logs, a comprehensive but noisy source of information about users’ dietary habits. We also introduce accompanying metrics to evaluate personal learning food preferences. The method generates and compares word embeddings to identify the parent food category of each food entry and then calculates the most popular. Our proposed approach identifies 82% of a user’s ten most frequently eaten foods. Our method is publicly available on (https://github.com/aametwally/LearningFoodPreferences).

Multiple sclerosis (MS) is a heterogeneous immune-mediated disorder of the central nervous system, and primary progressive MS (PPMS) is characterized by insidious injury independent of clinical relapses from disease onset. Predicting disease progression remains an unmet need across the MS disease spectrum. We evaluated statistical and machine learning (ML) models for predicting progression in 596 patients with PPMS from ORATORIO, a randomized, placebo (PBO)-controlled, phase 3 clinical study of ocrelizumab (OCR). Using baseline clinical, magnetic resonance imaging, and biomarker (neurofilament light chain [NfL]) data, we trained 6 ML models to predict a progression event (logistic regression, support vector machines, random forest, neural network, gradient boosting, and extreme gradient boosting) and the time to progression (Cox proportional hazards [PH]) on upper limb function, as measured by 20% confirmed worsening on the 9-hole peg test (9HPT). Predictive performance was similar across the six ML models (area under the curve = 0.63 to 0.67) and was modest using the Cox PH model (C-index = 0.76, an improvement over previous studies). In the Cox PH model, treatment arm, baseline 9HPT, neurologic functional system scores, and serum NfL levels were identified as factors associated with progression; model performance was similar across the OCR, PBO, and combined OCR + PBO treatment arms. Accounting for potential interactions with treatment arm resulted in similar predictive performance. To our knowledge, this is the first study to systematically evaluate ML models and statistical methods for predicting progression in PPMS using a well-characterized clinical trial cohort.

Accurate prediction of the host phenotype from a metagenomic sample and identification of the associated microbial markers are important in understanding potential host-microbiome interactions related to disease initiation and progression. We introduce PopPhy-CNN, a novel convolutional neural network (CNN) learning framework that effectively exploits phylogenetic structure in microbial taxa for host phenotype prediction. Our approach takes an input format of a 2D matrix representing the phylogenetic tree populated with the relative abundance of microbial taxa in a metagenomic sample. This conversion empowers CNNs to explore the spatial relationship of the taxonomic annotations on the tree and their quantitative characteristics in metagenomic data. We show the competitiveness of our model compared to other available methods using nine metagenomic datasets of moderate size for binary classification. With synthetic and biological datasets, we show the superior and robust performance of our model for multi-class classification. Furthermore, we design a novel scheme for feature extraction from the learned CNN models and demonstrate improved performance when the extracted features. PopPhy-CNN is a practical deep learning framework for the prediction of host phenotype with the ability of facilitating the retrieval of predictive microbial taxa.

Food allergy is usually difficult to diagnose in early life, and the inability to diagnose patients with atopic diseases at an early age may lead to severe complications. Numerous studies have suggested an association between the infant gut microbiome and development of allergy. In this work, we investigated the capacity of Long Short-Term Memory (LSTM) networks to predict food allergies in early life (0-3 years) from subjects' longitudinal gut microbiome profiles. Using the DIABIMMUNE dataset, we show an increase in predictive power using our model compared to Hidden Markov Model, Multi-Layer Perceptron Neural Network, Support Vector Machine, Random Forest, and LASSO regression. We further evaluated whether the training of LSTM networks benefits from reduced representations of microbial features. We considered sparse autoencoder for extraction of potential latent representations in addition to standard feature selection procedures based on Minimum Redundancy Maximum Relevance (mRMR) and variance prior to the training of LSTM networks. The comprehensive evaluation reveals that LSTM networks with the mRMR selected features achieve significantly better performance compared to the other tested machine learning models.