Object-centric learning aims to break down complex visual scenes into more manageable object representations, enhancing the understanding and reasoning abilities of machine learning systems toward the physical world. Recently, slot-based video models have demonstrated remarkable proficiency in segmenting and tracking objects, but they overlook the importance of the effective reasoning module. In the real world, reasoning and predictive abilities play a crucial role in human perception and object tracking; in particular, these abilities are closely related to human intuitive physics. Inspired by this, we designed a novel reasoning module called the Slot-based Time-Space Transformer with Memory buffer (STATM) to enhance the model's perception ability in complex scenes. The memory buffer primarily serves as storage for slot information from upstream modules, the Slot-based Time-Space Transformer makes predictions through slot-based spatiotemporal attention computations and fusion. Our experimental results on various datasets indicate that the STATM module can significantly enhance the capabilities of multiple state-of-the-art object-centric learning models for video. Moreover, as a predictive model, the STATM module also performs well in downstream prediction and Visual Question Answering (VQA) tasks. We will release our codes and data at https://github.com/intell-sci-comput/STATM.

Data-centric methods have shown great potential in understanding and predicting spatiotemporal dynamics, enabling better design and control of the object system. However, deep learning models often lack interpretability, fail to obey intrinsic physics, and struggle to cope with the various domains. While geometry-based methods, e.g., graph neural networks (GNNs), have been proposed to further tackle these challenges, they still need to find the implicit physical laws from large datasets and rely excessively on rich labeled data. In this paper, we herein introduce the conservation-informed GNN (CiGNN), an end-to-end explainable learning framework, to learn spatiotemporal dynamics based on limited training data. The network is designed to conform to the general conservation law via symmetry, where conservative and non-conservative information passes over a multiscale space enhanced by a latent temporal marching strategy. The efficacy of our model has been verified in various spatiotemporal systems based on synthetic and real-world datasets, showing superiority over baseline models. Results demonstrate that CiGNN exhibits remarkable accuracy and generalizability, and is readily applicable to learning for prediction of various spatiotemporal dynamics in a spatial domain with complex geometry.

Sparse distributions of seismic sensors and sources pose challenges for subsurface imaging, source characterization, and ground motion modeling. While large-N arrays have shown the potential of dense observational data, their deployment over extensive areas is constrained by economic and logistical limitations. Numerical simulations offer an alternative, but modeling realistic wavefields remains computationally expensive. To address these challenges, we develop a multi-conditional diffusion transformer for generating seismic wavefields without requiring prior geological knowledge. Our method produces high-resolution wavefields that accurately capture both amplitude and phase information across diverse source and station configurations. The model first generates amplitude spectra conditioned on input attributes and subsequently refines wavefields through iterative phase optimization. We validate our approach using data from the Geysers geothermal field, demonstrating the generation of wavefields with spatial continuity and fidelity in both spectral amplitude and phase. These synthesized wavefields hold promise for advancing structural imaging and source characterization in seismology.

Recent advances in foundation models have emphasized the need to align pre-trained models with specialized domains using small, curated datasets.Studies on these foundation models underscore the importance of low-data training and fine-tuning.This topic, well-known in natural language processing (NLP), has also gained increasing attention in the emerging field of scientific machine learning (SciML).To address the limitations of low-data training and fine-tuning, we draw inspiration from Heavy-Tailed Self-Regularization (HT-SR) theory, analyzing the shape of empirical spectral densities (ESDs) and revealing an imbalance in training quality across different model layers.To mitigate this issue, we adapt a recently proposed layer-wise learning rate scheduler, TempBalance, which effectively balances training quality across layers and enhances low-data training and fine-tuning for both NLP and SciML tasks.Notably, TempBalance demonstrates increasing performance gains as the amount of available tuning data decreases.Comparative analyses further highlight the effectiveness of TempBalance and its adaptability as an "add-on" method for improving model performance.

Generating realistic time series data is important for many engineering and scientific applications. Existing work tackles this problem using generative adversarial networks (GANs). However, GANs are unstable during training, and they can suffer from mode collapse. While variational autoencoders (VAEs) are known to be more robust to the these issues, they are (surprisingly) less considered for time series generation. In this work, we introduce Koopman VAE (KoVAE), a new generative framework that is based on a novel design for the model prior, and that can be optimized for either regular and irregular training data. Inspired by Koopman theory, we represent the latent conditional prior dynamics using a linear map. Our approach enhances generative modeling with two desired features: (i) incorporating domain knowledge can be achieved by leveraging spectral tools that prescribe constraints on the eigenvalues of the linear map; and (ii) studying the qualitative behavior and stability of the system can be performed using tools from dynamical systems theory. Our results show that KoVAE outperforms state-of-the-art GAN and VAE methods across several challenging synthetic and real-world time series generation benchmarks. Whether trained on regular or irregular data, KoVAE generates time series that improve both discriminative and predictive metrics. We also present visual evidence suggesting that KoVAE learns probability density functions that better approximate the empirical ground truth distribution.

Recent years have witnessed the promise of coupling machine learning methods and physical domain-specific insights for solving scientific problems based on partial differential equations (PDEs). However, being data-intensive, these methods still require a large amount of PDE data. This reintroduces the need for expensive numerical PDE solutions, partially undermining the original goal of avoiding these expensive simulations. In this work, seeking data efficiency, we design unsupervised pretraining for PDE operator learning. To reduce the need for training data with heavy simulation costs, we mine unlabeled PDE data without simulated solutions, and we pretrain neural operators with physics-inspired reconstruction-based proxy tasks. To improve out-of-distribution performance, we further assist neural operators in flexibly leveraging a similarity-based method that learns in-context examples, without incurring extra training costs or designs. Extensive empirical evaluations on a diverse set of PDEs demonstrate that our method is highly data-efficient, more generalizable, and even outperforms conventional vision-pretrained models. We provide our code at https://github.com/delta-lab-ai/data_efficient_nopt.

When solving partial differential equations (PDEs), classical numerical methods often require fine mesh grids and small time stepping to meet stability, consistency, and convergence conditions, leading to high computational cost. Recently, machine learning has been increasingly utilized to solve PDE problems, but they often encounter challenges related to interpretability, generalizability, and strong dependency on rich labeled data. Hence, we introduce a new PDE-Preserved Coarse Correction Network (P$^2$C$^2$Net) to efficiently solve spatiotemporal PDE problems on coarse mesh grids in small data regimes. The model consists of two synergistic modules: (1) a trainable PDE block that learns to update the coarse solution (i.e., the system state), based on a high-order numerical scheme with boundary condition encoding, and (2) a neural network block that consistently corrects the solution on the fly. In particular, we propose a learnable symmetric Conv filter, with weights shared over the entire model, to accurately estimate the spatial derivatives of PDE based on the neural-corrected system state. The resulting physics-encoded model is capable of handling limited training data (e.g., 3--5 trajectories) and accelerates the prediction of PDE solutions on coarse spatiotemporal grids while maintaining a high accuracy. P$^2$C$^2$Net achieves consistent state-of-the-art performance with over 50\% gain (e.g., in terms of relative prediction error) across four datasets covering complex reaction-diffusion processes and turbulent flows.

Three-dimensional (3D) modeling is currently a creative task that requires modelers with strong professional skills and background knowledge, especially in the field of 3D modeling of Chinese ancient architecture (CAA). At present, most of the studies on 3D CAA modeling are based on hard-coded constructive rules, which need completed, complex and formalized descriptions. We present a generative system bridging the gap between the Chinese text and 3D models that allows users to generate 3D models by natural language. First, a Bayesian network is learned from existing CAA data to provide relationships of different structural components. Second, by parsing the Chinese text inputted by the user, key components of the CAA will be determined; and other matched structural components will be calculated by inferencing the trained Bayesian network. Third, the synthesis of all components is achieved by a proposed placement optimizing algorithm. Finally, we evaluate the effectiveness of the trained Bayesian network and demonstrate the application to generate 3D CAA model rapidly from the Chinese text.

The existing 3D scene layout methods, which mainly focus on indoor scenes, are limited in outdoor applications. In this paper, the example-based modeling method is introduced into the outdoor modeling, and an automatic layout optimization method for outdoor scenes is proposed. As an application platform, the sketch-based 3D model retrieval and assemble system is realized as well. Different from the current methods, firstly, we adopt an improved manifold sorting algorithm in sketch retrieval method, which can get the 3D models rapidly; secondly, according to particular properties of outdoor architectures, specialized energy constraints are proposed, which defines the energy function that meets the functional and aesthetic needs; thirdly, the scene achieves automatic layout by adopting simulated annealing algorithm. By stepped asymptotic optimization strategy and random jumps, we can avoid falling into constraint conflicts and local optimal traps. Experimental results show the robustness and effectiveness of our algorithm in different scenes. Our algorithm has been applied in the actual development of game scenes.