Multi-modal time series analysis has recently emerged as a prominent research area, driven by the increasing availability of diverse data modalities, such as text, images, and structured tabular data from real-world sources. However, effective analysis of multi-modal time series is hindered by data heterogeneity, modality gap, misalignment, and inherent noise. Recent advancements in multi-modal time series methods have exploited the multi-modal context via cross-modal interactions based on deep learning methods, significantly enhancing various downstream tasks. In this tutorial and survey, we present a systematic and up-to-date overview of multi-modal time series datasets and methods. We first state the existing challenges of multi-modal time series analysis and our motivations, with a brief introduction of preliminaries. Then, we summarize the general pipeline and categorize existing methods through a unified cross-modal interaction framework encompassing fusion, alignment, and transference at different levels (i.e., input, intermediate, output), where key concepts and ideas are highlighted. We also discuss the real-world applications of multi-modal analysis for both standard and spatial time series, tailored to general and specific domains. Finally, we discuss future research directions to help practitioners explore and exploit multi-modal time series. The up-to-date resources are provided in the GitHub repository. https://github.com/UConn-DSIS/Multi-modal-Time-Series-Analysis.

Jointly learning from multiple datasets can help building versatile intelligent systems yet may give rise to serious concerns of data privacy and model selection. Specifically, on the one hand, these datasets can be distributed at various local clients, who may not be willing or do not ought to share data with each other. On the other hand, it is unrealistic to choose a model architecture that can well suit the disparate patterns and distributions carried by the various datasets in a priori. Whereas many works in federated learning [1] and neural architecture search [2] have been proposed to address one of the two concerns, very few have attempted the both. To close the gap, in this paper we deliver a framework, termedMulti-Granular Federated Neural Architecture Search(MGFNAS), to enable the automation of model architecture search in a federated and thus privacy-preserved setting. We argue that our MGFNAS framework is general in the sense that it does not impose any restriction on the search space or strategy, such that most existing neural architecture search techniques can be readily implemented in. The main idea of our framework is to search the optimal neural network architecture in two levels of granularity, enabling the neural-operator-basedmicro-levelsearch and the cell-basedmacro-levelsearch. The main challenge of implementing our framework lies in the fact that, due to the decentralized nature, the local architectures searched by multiple clients can differ drastically in order to fit their own datasets, while a general method to form the global model by aggregating the local architectures in both micro and macro levels is missing. To solve the issue, we propose a novel aggregation function, named Network Architecture Probabilistic Aggregation (NAPA). The key idea of our NAPA function is to treat the network architectures as graphs, of which the sub-graph structures being frequently appeared across multiple clients are modeled by probabilistic distributions. At each round, a global model is formed by sampling from those distributions in an exploration-exploitation fashion. Extensive experiments are carried out, and the results substantiate the viability and effectiveness of our proposed framework.