Large-scale scientific simulations require significant resources to generate high-resolution time-varying data (TVD). While super-resolution is an efficient post-processing strategy to reduce costs, existing methods rely on a large amount of HR training data, limiting their applicability to diverse simulation scenarios. To address this constraint, we proposed CD-TVD, a novel framework that combines contrastive learning and an improved diffusion-based super-resolution model to achieve accurate 3D super-resolution from limited time-step high-resolution data. During pre-training on historical simulation data, the contrastive encoder and diffusion super-resolution modules learn degradation patterns and detailed features of high-resolution and low-resolution samples. In the training phase, the improved diffusion model with a local attention mechanism is fine-tuned using only one newly generated high-resolution timestep, leveraging the degradation knowledge learned by the encoder. This design minimizes the reliance on large-scale high-resolution datasets while maintaining the capability to recover fine-grained details. Experimental results on fluid and atmospheric simulation datasets confirm that CD-TVD delivers accurate and resource-efficient 3D super-resolution, marking a significant advancement in data augmentation for large-scale scientific simulations. The code is available at https://github.com/Xin-Gao-private/CD-TVD.

Talking head synthesis is to synthesize a lip-synchronized talking head video using audio. Recently, the capability of NeRF to enhance the realism and texture details of synthesized talking heads has attracted the attention of researchers. However, most current NeRF methods based on audio are exclusively concerned with the rendering of frontal faces. These methods are unable to generate clear talking heads in novel views. Another prevalent challenge in current 3D talking head synthesis is the difficulty in aligning acoustic and visual spaces, which often results in suboptimal lip-syncing of the generated talking heads. To address these issues, we propose Neural Radiance Field with 3D Prior Aided Audio Disentanglement for Talking Head Synthesis (NeRF-3DTalker). Specifically, the proposed method employs 3D prior information to synthesize clear talking heads with free views. Additionally, we propose a 3D Prior Aided Audio Disentanglement module, which is designed to disentangle the audio into two distinct categories: features related to 3D awarded speech movements and features related to speaking style. Moreover, to reposition the generated frames that are distant from the speaker’s motion space in the real space, we have devised a local-global Standardized Space. This method normalizes the irregular positions in the generated frames from both global and local semantic perspectives. Through comprehensive qualitative and quantitative experiments, it has been demonstrated that our NeRF3DTalker outperforms state-of-the-art in synthesizing realistic talking head videos, exhibiting superior image quality and lip synchronization. Project page: https://nerf-3dtalker.github.io/NeRF-3Dtalker/.

Implicit neural representation (INR) has now been thrust into the limelight with its flexibility in high-fidelity flow field reconstruction tasks. However, the lack of standard benchmarking datasets and the grid independence assumption for INR-based methods hinder progress and adoption in real-world simulation scenarios. Moreover, naive adoptions of existing INR frameworks suffer from limited accuracy in capturing fine-scale structures and spatiotemporal dynamics. Tacking these issues, we first introduce HFR-Beach, a 5.4 TB public large-scale CFD dataset with 33,600 unsteady 2D and 3D vector fields for reconstructing high-fidelity flow fields. We further present PEINR, a physics-enhanced INR framework, to enrich the flow fields by concurrently enhancing numerical-precision and grid-resolution. Specifically, PEINR is mainly composed of physical encoding and transformer-based spatiotemporal fuser (TransSTF). Physical encoding decouples temporal and spatial components, employing Gaussian coordinate encoding and localized encoding techniques to capture the nonlinear characteristics of spatiotemporal dynamics and the stencil discretization of spatial dimensions, respectively. TransSTF fuses both spatial and temporal information via transformer for capturing long-range temporal dependencies. Qualitative and quantitative experiments and demonstrate that PEINR outperforms state-of-the-art INR-based methods in reconstruction quality.

Accurate analysis of time-dependent flow fields generated by numerical simulations requires effective interpretation of temporal information. Visualizations offer an exceptional ability to convey complex data and have been widely used for such analysis. However, current research mainly focuses on flow features at individual time steps, lacking exploration of how these features evolve over time, leading to insufficient investigation and understanding of fluid motion laws. To overcome this limitation, this study proposes BiFTVis, an interactive visualization framework that enhances the complete flow field analysis pipeline. The framework includes a volume rendering module that represents flow features extracted using the Q criterion method. An additional forward tracking module is provided to track feature events by utilizing a graph optimization-based feature tracking algorithm. A timeline-based visual encoding method has been designed to convey multiple feature events along the time axis, facilitating the examination of events in specific time periods. Furthermore, a novel radial layout glyph along with a Directed Acyclic Graph has been designed for encoding multi-facet feature attributes, enabling fast identification of various feature types and reverse tracking analysis of feature events in a backward tracking module. BiFTVis has been evaluated through a case study, demonstrating its effectiveness in promoting new insights into potential causes of feature events and avoiding certain feature event occurrences, ultimately enhancing flow field analysis.

To address the challenges of accurately predicting vehicle trajectories and prioritizing driving tasks in complex situations such as lane changing or highway ramp merging for autonomous vehicles, this paper introduces a deep reinforcement learning (DRL) merging control method called DRLI-P (DRL for Trajectory Prediction and Task Importance Network Fusion). DRLI-P integrates an LSTM-based vehicle trajectory prediction network with a rule-based task importance network (TIN). The TIN assesses the importance of vehicle action rules and complex driving tasks, alleviating problems associated with sparse reward distribution in DRL and improving sampling efficiency. The LSTM uses historical driving data to predict vehicle trajectories and constructs a state space to mitigate slow training speeds and reduced sensitivity to single state parameter changes due to high state dimensionality in multi-vehicle scenarios. A multi-category weighted reward function is developed that focuses on critical driving features such as target distance, vehicle motion information, and trajectory predictions. The proposed merging control method is applied to three leading DRL algorithms: DDPG, TD3, and SAC, followed by simulation experiments in the CARLA environment. The results show that the DRLI-P method significantly improves the convergence speed and performance of all three algorithms, with the most notable improvement seen in the SAC algorithm, thereby increasing the safety, efficiency, and convenience of DRL algorithms for merging control.

Porous materials (e.g., weathered stone, industrial coatings) exhibit complex optical effects due to their micrograin and pore structures, posing challenges for photorealistic rendering. Explicit geometry models struggle to characterize their micrograin distributions at microscopic scales, while single-scattering microfacet model fails to accurately capture the multiple-scattering effects and causes energy non-conservation artifacts, manifesting as unrealistic luminance decay. We propose an enhanced micrograin BSDF model that accurately accounts for multiple scattering. First, we introduce a visible normal distribution function (VNDF) sampling method via rejection sampling. Building on VNDF sampling, we derive a position-free microsurface formulation incorporating both inter-micrograin and micrograin-to-base interactions. Furthermore, we propose a practical random walk method to simulate microsurface scattering, which accurately solves the derived formulation. Our micrograin BSDF model effectively eliminates the energy loss artifacts inherent in the previous model while significantly reducing noise, providing a physically accurate yet artistically controllable solution for rendering porous materials.

Semantic segmentation of 3D point clouds is important for many applications, such as autonomous driving. To train semantic segmentation models, labeled point cloud segmentation datasets are essential. Meanwhile, point cloud labeling is time-consuming for annotators, which typically involves tuning the camera viewpoint and selecting points by lasso. To reduce the time cost of point cloud labeling, we propose a viewpoint recommendation approach to reduce annotators' labeling time costs. We adapt Fitts' law to model the time cost of lasso selection in point clouds. Using the modeled time cost, the viewpoint that minimizes the lasso selection time cost is recommended to the annotator. We build a data labeling system for semantic segmentation of 3D point clouds that integrates our viewpoint recommendation approach. The system enables users to navigate to recommended viewpoints for efficient annotation. Through an ablation study, we observed that our approach effectively reduced the data labeling time cost. We also qualitatively compare our approach with previous viewpoint selection approaches on different datasets.

With the recent developments in computational science and HPC technology, large-scale numerical simulations have become common in various scientific and technological fields. The output volume data from these simulations have also become larger and more complex, creating a problem for the time-consuming input/output to/from the HPC storage system. To solve this problem, in-situ visualization has been used. However, the output data for posterior analysis is usually a large set of image data obtained from the visualization, and there is a lack of interactivity compared to the conventional analysis, which loads the volume data from the storage, after the simulation, and executes interactive visual exploration. To compensate for these problems, in-situ visualization often places multiple viewpoints in the simulation space and generates images from all of them. However, this may result in a huge number of images, and as a result, this can require time and effort to locate important visualization images that can provide clues to obtain knowledge during the analysis. To solve this problem, this study estimates the regions where important changes occur in the simulation, based on information entropy calculated from the visualization images, and generates a sequence of animated images focusing on these regions. In-situ visualization has widely been recognized as an effective approach for analyzing large-scale simulation outputs from modern HPC systems by reducing the inherent I/O bottleneck problem. However, batch-based in-situ visualization, such as the image- and video-based approaches, can produce large amounts of rendering results for the subsequent offline visual analysis. Therefore, this can make it difficult to gain rapid insight into the simulation results during post-hoc visual analysis. To minimize this problem, we have worked on a smart visualization approach focusing on extracting a set of images that may facilitate the rapid understanding of the underlying simulated phenomena as an alternative to accelerate the process of obtaining scientific knowledge. In this work, we present a method for automatically adjusting the camera focus point and zoom level during in-situ visualization in an attempt to obtain the most suitable rendering images for facilitating visual analysis. We integrated the proposed method with the existing in-situ smooth camera path estimation framework, for evaluation purposes, and used two CFD simulation codes and two HPC systems (x86-based server system and Arm-based Fugaku supercomputer) for the evaluations. We obtained encouraging results from the preliminary evaluations, and we are planning further improvements by working closely with domain expert collaborators.

Energy efficiency has become a serious concern when running applications on HPC systems. Although these systems were designed to mainly run simulation codes as fast as possible, due to the ever-increasing size of the simulation outputs, the in situ visualization has gained increasing attention. In situ visualization uses the same HPC system to execute a part or even the entire visualization processing, and there are currently a variety of tools and libraries, that facilitate domain scientists to integrate them with their simulation codes. Among different approaches, image- and video-based in situ visualization has been widely adopted as an effective approach for the subsequent offline visual analysis. In this approach, a large number of renderings are required at every visualization time step and can consume a considerable computational resource. Fugaku adopted PowerAPI which enables the users to set the power mode for their jobs. However, simulation and visualization codes may have different processing behaviors requiring different power settings for obtaining the most energy-efficient runnings. In this work, we tried to shed light on the energy efficiency of the visualization portion that was not considered before. We investigated the computational cost and energy consumption of some rendering techniques by using the PowerAPI and KVS (Kyoto Visualization System) on the Fugaku, and hope that the obtained findings will be useful for potential users looking to run in situ visualization on the Fugaku and other PowerAPI-enabled HPC systems.

Talking face synthesis driven by audio is one of the current research hotspots in the fields of multidimensional signal processing and multimedia. Neural Radiance Field (NeRF) has recently been brought to this research field in order to enhance the realism and 3D effect of the generated faces. However, most existing NeRF-based methods either burden NeRF with complex learning tasks while lacking methods for supervised multimodal feature fusion, or cannot precisely map audio to the facial region related to speech movements. These reasons ultimately result in existing methods generating inaccurate lip shapes. This paper moves a portion of NeRF learning tasks ahead and proposes a talking face synthesis method via NeRF with attention-based disentanglement (NeRF-AD). In particular, an Attention-based Disentanglement module is introduced to disentangle the face into Audio-face and Identity-face using speech-related facial action unit (AU) information. To precisely regulate how audio affects the talking face, we only fuse the Audio-face with audio feature. In addition, AU information is also utilized to supervise the fusion of these two modalities. Extensive qualitative and quantitative experiments demonstrate that our NeRF-AD outperforms state-of-the-art methods in generating realistic talking face videos, including image quality and lip synchronization. To view video results, please refer to https://xiaoxingliu02.github.io/NeRF-AD/.

In the field of object detection bounding box regression, IoU (Intersection over Union) is a commonly used evaluation metric for measuring the overlap between predicted and ground truth bounding boxes. One limitation of IoU, however, is that the gradient goes to zero when there is no intersection between bounding boxes. Recent evaluation metrics have been developed to overcome this problem by incorporating penalty terms and other techniques for improving IoU. Despite these advances, existing methods still suffer to some degree from the problems of small gradients and slow rate of convergence. In this paper, a new evaluation method named LIoU (Location IoU) is proposed to address these issues. It can maximize the shared area of the intersection when used as the loss function and incorporate a variable gradient parameter to fit to datasets of different sizes. The experimental results indicate that we achieve 0.27% and 3.48% accuracy improvements respectively in SSD and YOLOv5 network on the VOC dataset, and 2.23% in the YOLOv5 network on the COCO dataset. The results of this study highlight the effectiveness of LIoU in improving the performance of object detection models.

Understanding driving scenarios is a critical issue in autonomous driving, as it allows developers to create more effective and reliable autonomous driving systems. However, the complexity and dynamics of driving scenarios present a significant challenge. In this paper, we utilize scene graphs to encode the semantics of driving scenarios and propose a spatio-temporal visual analytics system to support scenario comprehension. The system provides a multi-level visualization of autonomous driving scenarios, including single-frame semantics, time-varying visual summaries, and scenario distribution. With a subgraph matching algorithm, the system supports users in interactively mining and comparing driving scenarios of interest. Users can explore, analyze, compare, summarize, and draw conclusions about various scenarios through our system. We demonstrate the effectiveness of the system through case studies. Our system can help users identify meaningful scenarios and corner cases, thus providing support for autonomous driving simulation and data augmentation.

Traditional deep learning algorithms assume that all data is available during training, which presents challenges when handling large-scale time-varying data. To address this issue, we propose a data reduction pipeline called knowledge distillation-based implicit neural representation (KD-INR) for compressing large-scale time-varying data. The approach consists of two stages: spatial compression and model aggregation. In the first stage, each time step is compressed using an implicit neural representation with bottleneck layers and features of interest preservation-based sampling. In the second stage, we utilize an offline knowledge distillation algorithm to extract knowledge from the trained models and aggregate it into a single model. We evaluated our approach on a variety of time-varying volumetric data sets. Both quantitative and qualitative results, such as PSNR, LPIPS, and rendered images, demonstrate that KD-INR surpasses the state-of-the-art approaches, including learning-based (i.e., CoordNet, NeurComp, and SIREN) and lossy compression (i.e., SZ3, ZFP, and TTHRESH) methods, at various compression ratios ranging from hundreds to ten thousand.

Traffic congestion is becoming increasingly severe as a result of urbanization, which not only impedes people’s ability to travel but also hinders the economic development of cities. Modelling the correlation between congestion and its influencing factors using machine learning methods make it possible to quickly identify congested road segments. Due to the intrinsic black-box character of machine learning models, it is difficult for experts to trust the decision results of road congestion prediction models and understand the significance of congestion-causing factors. In this paper, we present a model interpretability method to investigate the potential causes of traffic congestion and quantify the importance of various influencing factors using the SHAP method. Due to the multidimensionality of these factors, it can be challenging to visually represent the impact of all factors. In response, we propose TCEVis, an interactive visual analytics system that enables multi-level exploration of road conditions. Through three case studies utilizing actual data, we demonstrate that the TCEVis system offers advantages for assisting traffic managers in analyzing the causes of traffic congestion and elucidating the significance of various influencing factors.

Recently, controlling air pollution has become increasingly significant due to its impact on our health and daily lives. To prevent and control pollution, it is crucial to trace its source. Many researches have been developed for tracing the source of pollution. However, traditional methods using large-scale simulations need a large number of computation resources and time-consuming. In addition, traditional traceability algorithms do not consider topographic factors, which can cause a certain amount of errors. To resolve above problems, an interactive visual analytics system for pollutant traceability is proposed. In our method, instead of three-dimensional field data, only two-dimensional grid data is enough to track pollution sources in real time. Furthermore, our method can further improve precision through considering topographic factors, which are usually ignored by existing methods. Finally, the possible pollution sources are also identified in our method. This is achieved through analysis of changes in pollutant concentration and the distribution of man-made emission sources. In order to verify the effectiveness of this method, we propose a series of application examples to comprehensively analyze the sources of pollutants.

Nowadays, social networks are becoming a popular way of analyzing tourist behavior, thanks to the digital traces left by travelers during their stays on these networks. The massive amount of data generated; by the propensity of tourists to share comments and photos during their trip; makes it possible to model their journeys and analyze their behavior. Predicting the next movement of tourists plays a key role in tourism marketing to understand demand and improve decision support.In this paper, we propose a method to understand and to learn tourists' movements based on social network data analysis to predict future movements. The method relies on a machine learning grammatical inference algorithm. A major contribution in this paper is to adapt the grammatical inference algorithm to the context of big data. Our method produces a hidden Markov model representing the movements of a group of tourists. The hidden Markov model is flexible and editable with new data. The capital city of France, Paris is selected to demonstrate the efficiency of the proposed methodology.

The development of supercomputers has successfully helped us to carry on complicated simulation with exploded size of dataset. For visualizing such kind of large-scale dataset, reducing the data size by using compression methods is one of the most useful approach. Moreover, parallelization of compression algorithm can greatly improve the efficiency and resolve the limitation of memory size. However, in parallel compression algorithm, interprocessor communication is indispensable, while it is also a bottleneck problem, especially for the general cases that the number of processors is not power-of-two. Parallel POD (proper orthogonal decomposition) compression algorithm is such an example, the number of time steps must be power-of-two for the binary swap scheme. A method that can fully resolve this problem with low computational cost will be very popular. In this paper, we proposed such an approach called 2-3-4 combination approach, which can be simply implemented and also reach high performance of parallel computing algorithms. Furthermore, our method can obtain the best balance among all parallel computing processors. This is achieved by transferring the non-power-of-two problem into power-of-two problem to fully use the best balance feature of binary swap method. We evaluate our approach through applying it to the parallel POD compression algorithm on the K computer.

It is difficult to carry out visualization of the large-scale time-varying data directly, even with the supercomputers. Data compression and ROI (Region of Interest) detection are often used to improve efficiency of the visualization of numerical data. It is well known that the Run Length encoding is a good technique to compress the data where the same sequence appeared repeatedly, such as an image with little change, or a set of smooth fluid data. Another advantage of Run Length encoding is that it can be applied to every dimension of data separately. Therefore, the Run Length method can be implemented easily as a parallel processing algorithm. We proposed two different Run Length based methods. When using the Run Length method to compress a data set, its size may increase after the compression if the data does not contain many repeated parts. We only apply the compression for the case that the data can be compressed effectively. By checking the compression ratio, we can detect ROI. The effectiveness and efficiency of the proposed methods are demonstrated through comparing with several existing compression methods using different sets of fluid data.