Abstract Using 3D Gaussian splatting to reconstruct large‐scale aerial scenes from ultra‐high‐resolution images is still a challenge problem because of two memory bottlenecks ‐ excessive Gaussian primitives and the tensor sizes for ultra‐high‐resolution images. In this paper, we propose a task partitioning algorithm that operates in both object and image space to generate a set of small‐scale subtasks. Each subtask's memory footprints is strictly limited, enabling training on a single high‐end consumer‐grade GPU. More specifically, Gaussian primitives are clustered into blocks in object space, and the input images are partitioned into sub‐images according to the projected footprints of these blocks. This dual‐space partitioning significantly reduces training memory requirements. During subtask training, we propose a depth comparison method to generate a mask map for each sub‐image. This mask map isolates pixels primarily contributed by the Gaussian primitives of the current subtask, excluding all other pixels from training. Experimental results demonstrate that our method successfully achieves large‐scale aerial scene reconstruction using 9K resolution images on a single RTX 4090 GPU. The novel views synthesized by our method retain significantly more details than those from current state‐of‐the‐art methods.

Keypoint detection and matching have garnered significant attention, yet remain challenging in low-light environments. Most current studies follow anenhance-then-detectpipeline, which consists of independent enhancers and detectors. While the enhancer focuses on improving the visual quality of low-light images to satisfy human perception standards, the detector prioritizes detection accuracy for machine vision tasks. The unaligned optimization objectives of the enhancer and detector overlook the gap between human and machine vision and lead to sub-optimal performance in low-light keypoint detection. To tackle this problem, a jointenhance-and-detectpipeline is proposed to unify the optimization objectives of enhancement and detection by regarding the improvement of keypoint detection accuracy with enhanced features under machine vision standards. Specifically, we propose a low-light keypoint detection network named DeRFeat, which learns a degradation-equivariant representation between normal and dark domains using AutoEncoding transformation and domain descriptor similarity constraints to indirectly enhance the features from the encoder in the training stage. Then, DeRFeat guides the shared encoder to obtain the degradation-equivariant representations from dark images in the inference stage. With the dark degradation predictions, the encoder is capable of generating equivariant representations between normal and dark domains. The proposed domain descriptor similarity module further aids the encoder in mitigating the impact of dark degradation factors, enabling local descriptors to acquire undisturbed representations. Moreover, a coarse-to-fine point selection strategy is proposed to provide reliable prior keypoints for a globally optimal descriptor construction. Experimental results on four benchmark datasets demonstrate that the proposed method significantly outperforms state-of-the-art methods under varying low-light conditions.

Federated learning (FL) is an emerging approach for training machine learning models collaboratively while preserving data privacy. The need for privacy protection makes it difficult for FL models to achieve global transparency and explainability. To address this limitation, we incorporate logic-based explanations into FL by proposing the Logical Reasoning-based eXplainable Federated Learning (LR-XFL) approach. Under LR-XFL, FL clients create local logic rules based on their local data and send them, along with model updates, to the FL server. The FL server connects the local logic rules through a proper logical connector that is derived based on properties of client data, without requiring access to the raw data. In addition, the server also aggregates the local model updates with weight values determined by the quality of the clients’ local data as reflected by their uploaded logic rules. The results show that LR-XFL outperforms the most relevant baseline by 1.19%, 5.81% and 5.41% in terms of classification accuracy, rule accuracy and rule fidelity, respectively. The explicit rule evaluation and expression under LR-XFL enable human experts to validate and correct the rules on the server side, hence improving the global FL model’s robustness to errors. It has the potential to enhance the transparency of FL models for areas like healthcare and finance where both data privacy and explainability are important.

3D Gaussian Splatting (3DGS) has recently gained increasing attention in novel-view scene synthesis. However, it requires millions of 3D Gaussian spheres to achieve high-quality rendered images, leading to substantial GPU resource demands for training and rendering. This paper has developed an efficient framework to address the challenges faced by 3DGS. 1) We propose an asymmetric mixed-precision quantization strategy to efficiently quantize and dequantize the parameters of 3D Gaussian Spheres (excluding position and opacity) and utilized the Straight Through Estimator method to address the issue of non-backpropagatable gradients post-quantization. This approach significantly reduces GPU memory usage and model storage space. 2) Using statistical methods, we identify optimal gradient thresholds to enhance the densification algorithm of 3D Gaussian spheres, thereby further improving performance. 3) To address the speed bottlenecks in parallel processing of low-precision data using CUDA, we introduce a fast atomic operation method, increasing training speed tenfold. Overall, we validate the effectiveness of our framework across various datasets. It reduces GPU usage by 50% during training, decreases GPU usage threefold during rendering, halves the model storage space and maintains image quality comparable to 3DGS.

Federated learning (FL) is an emerging paradigm of collaborative machine learning that preserves user privacy while building powerful models. Nevertheless, due to the nature of open participation by self-interested entities, it needs to guard against potential misbehaviours by legitimate FL participants. FL verification techniques are promising solutions for this problem. They have been shown to effectively enhance the reliability of FL networks and build trust among participants. Verifiable FL has become an emerging topic of research that has attracted significant interest from the academia and the industry alike. Currently, there is no comprehensive survey on the field of verifiable federated learning, which is interdisciplinary in nature and can be challenging for researchers to enter into. In this paper, we bridge this gap by reviewing works focusing on verifiable FL. We propose a novel taxonomy for verifiable FL covering both centralised and decentralised settings, summarise the commonly adopted performance evaluation approaches, and discuss promising directions towards a versatile verifiable FL framework.

Abstract In this paper, we propose a novel second‐order explicit midpoint method to address the issue of energy loss and vorticity dissipation in Eulerian fluid simulation. The basic idea is to explicitly compute the pressure gradient at the middle time of each time step and apply it to the velocity field after advection. Theoretically, our solver can achieve higher accuracy than the first‐order solvers at similar computational cost. On the other hand, our method is twice and even faster than the implicit second‐order solvers at the cost of a small loss of accuracy. We have carried out a large number of 2D, 3D and numerical experiments to verify the effectiveness and availability of our algorithm.

Simulating shadow interactions between real and virtual objects is important for augmented reality (AR), in which accurately and efficiently detecting real shadows from live videos is a crucial step. Most of the existing methods are capable of processing only scenes captured under a fixed viewpoint. In contrast, this article proposes a new framework for shadow detection in live outdoor videos captured under moving viewpoints. The framework splits each frame into a tracked region, which is the region tracked from the previous video frame through optical flow analysis, and an emerging region, which is newly introduced into the scene due to the moving viewpoint. The framework subsequently extracts features based on the intensity profiles surrounding the boundaries of candidate shadow regions. These features are then utilized to both correct erroneous shadow boundaries for the tracked region and to detect shadow boundaries for the emerging region by a Bayesian learning module. To remove spurious shadows, spatial layout constraints are further considered for emerging regions. The experimental results demonstrate that the proposed framework outperforms the state-of-the-art shadow tracking and detection algorithms on a variety of challenging cases in real time, including shadows on backgrounds with complex textures, nonplanar shadows, fast-moving shadows with changing typologies, and shadows cast by nonrigid objects. The quantitative experiments show that our method outperforms the best existing method, achieving a 33.3% increase in the average$F_{measure}$on a self-collected database. Coupled with an image-based shadow-casting method, the proposed framework generates realistic shadow interaction results. This capability will be particularly beneficial for supporting AR applications.

Dementia is one of the major causes of disability and dependency among older people. Early detection is the key for preserving the quality of life of the patients and reducing caring costs. The Clock Drawing Test (CDT) is commonly used by clinicians to screen for early signs of dementia. We build an automated CDT that runs on mobile platforms, enabling convenient and frequent self-monitoring and testing at minimal costs. Our system combines both a spatial-temporal approach and a purely image-based deep learning approach to analyze and evaluate the hand-drawn clocks based on established clinical criteria. Our system produces scores that are highly correlated with expert human raters.

The quarterly financial statement, or Form 10-Q, is one of the most frequently required filings for US public companies to disclose financial and other important business information. Due to the massive volume of 10-Q filings and the enormous variations in the reporting format, it has been a long-standing challenge to retrieve item-specific information from 10-Q filings that lack machine-readable hierarchy. This paper presents a solution for itemizing 10-Q files by complementing a rule-based algorithm with a Convolutional Neural Network (CNN) image classifier. This solution demonstrates a pipeline that can be generalized to a rapid data retrieval solution among a large volume of textual data using only typographic items. The extracted textual data can be used as unlabeled content-specific data to train transformer models (e.g., BERT) or fit into various field-focus natural language processing (NLP) applications.

Population aging is becoming an increasingly important issue around the world. As people live longer, they also tend to suffer from more challenging medical conditions. Currently, there is a lack of a holistic technology-powered solution for providing quality care at affordable cost to patients suffering from co-morbidity. In this paper, we demonstrate a novel AI-powered solution to provide early detection of the onset of Dementia + Parkinson's disease (DPD) co-morbidity, a condition which severely limits a senior's ability to live actively and independently. We investigate useful in-game behaviour markers which can support machine learning-based predictive analytics on seniors' risk of developing DPD co-morbidity.

Abstract Physically based deformation simulation has been studied for many years in computer graphics. In order to simulate more complex geometric models and better meet the designer's requirements, many anisotropic approaches have been proposed in recent years. However, most of the approaches focus on simulating orthotropic models. In comparison with orthotropic models, nonorthotropic ones allow the objects to have anisotropic behaviors along nonorthogonal directions. In this paper, we introduce an improved approach to simulate nonorthotropic geometric models under large deformation. The improvements are mainly twofold. First, a frame field is specified on a given undeformed object, that is, each point of the object is equipped with a frame. In each local frame, we construct three independent vectors and form a nonorthogonal coordinate. Second, we design the deformation properties along each axis in the local nonorthogonal coordinate to get a local constitutive model. The final nonorthotropic model is generated by transforming the designed model from local nonorthogonal coordinates to the global standard Cartesian coordinate. To improve the stability, we introduce a time‐varying method to simultaneously track the local coordinates reorientation by pushing forward the original frame field to the deformed frame field. Experiments show that the deformation simulation using the designed nonorthotropic models exhibits anisotropic behaviors along different directions and are more stable than previous methods.

We propose an automatic framework to recover the illumination of indoor scenes based on a single RGB-D image. Unlike previous works, our method can recover spatially varying illumination without using any lighting capturing devices or HDR information. The recovered illumination can produce realistic rendering results. To model the geometry of the visible and invisible parts of scenes corresponding to the input RGB-D image, we assume that all objects shown in the image are located in a box with six faces and build a planar-based geometry model based on the input depth map. We then present a confidence-scoring based strategy to separate the light sources from the highlight areas. The positions of light sources both in and out of the camera's view are calculated based on the classification result and the recovered geometry model. Finally, an iterative procedure is proposed to calculate the colors of light sources and the materials in the scene. In addition, a data-driven method is used to set constraints on the light source intensities. Using the estimated light sources and geometry model, environment maps at different points in the scene are generated that can model the spatial variance of illumination. The experimental results demonstrate the validity and flexibility of our approach.

We propose a new path planning approach for crowd evacuation in buildings. Based on the crowd distribution at current moment, our approach is capable of predicting congestion at intersections in the near future so that a more reasonable path can be achieved than previous method. We also introduce a stage-wise path planning mechanism to adjust routes to respond to the dynamic changes of crowd distribution. Experimental results on several evacuation scenarios indicate that our method can reduce congestion comparing with previous method so that more reasonable evacuation routes can be achieved and the evacuation time is shortened.

In this paper, we present a data mining based algorithm to automatically locate performance bottlenecks at algorithm level for a complex rendering system. The basic idea is to treat the bottleneck identification problem as a variable importance analysis problem from a large volume of performance data which is generated by collecting the time costs under different combinations of algorithm level parameters. Based on the performance data set, random forest is adopted to conduct the variable importance ranking task. We also note an important fact that there might no performance bottleneck exists in the scope of the whole rendering system, but it is likely that bottlenecks could be found under some specific conditions. Thus we propose a bottleneck analysis tree to split the parameter space into many subspaces in which performance bottlenecks can be identified.

Abstract We introduce a new biorthogonal wavelet approach to creating a water‐tight surface defined by an implicit function, from a finite set of oriented points. Our approach aims at addressing problems with previous wavelet methods which are not resilient to missing or nonuniformly sampled data. To address the problems, our approach has two key elements. First, by applying a three‐dimensional partial integration, we derive a new integral formula to compute the wavelet coefficients without requiring the implicit function to be an indicator function. It can be shown that the previously used formula is a special case of our formula when the integrated function is an indicator function. Second, a simple yet general method is proposed to construct smooth wavelets with small support. With our method, a family of wavelets can be constructed with the same support size as previously used wavelets while having one more degree of continuity. Experiments show that our approach can robustly produce results comparable to those produced by the Fourier and Poisson methods, regardless of the input data being noisy, missing or nonuniform. Moreover, our approach does not need to compute global integrals or solve large linear systems.

Abstract Gradient‐domain compositing has been widely used to create a seamless composite with gradient close to a composite gradient field generated from one or more registered images. The key to this problem is to solve a Poisson equation, whose unknown variables can reach the size of the composite if no region of interest is drawn explicitly, thus making both the time and memory cost expensive in processing multi‐megapixel images. In this paper, we propose an approximate projection method based on biorthogonal Multiresolution Analyses (MRA) to solve the Poisson equation. Unlike previous Poisson equation solvers which try to converge to the accurate solution with iterative algorithms, we use biorthogonal compactly supported curl‐free wavelets as the fundamental bases to approximately project the composite gradient field onto a curl‐free vector space. Then, the composite can be efficiently recovered by applying a fast inverse wavelet transform. Considering an n‐pixel composite, our method only requires 2n of memory for all vector fields and is more efficient than state‐of‐the‐art methods while achieving almost identical results. Specifically, experiments show that our method gains a 5× speedup over the streaming multigrid in certain cases.

In this paper, we present a novel method to simulate the influences of small social group on pedestrian's behaviors under emergency situations. Our method is built on an important observation that the relationships between group members are usually different and even mutual relationships between group members might be asymmetric. Two phenomena can be produced by our method based on this observation. The first is group aggregation phenomenon which means that pedestrians tend to stay closer to their socially close group members than socially distant members. The second phenomenon is the complicated process of searching for lost members which is modeled as a cost-based function in our method. This function will guide pedestrians to make many decisions like whether to look for the lost members, who will be searched for, who will conduct the search as well as when to abort the search. The experimental results show that our method can produce very real social group behaviors under emergency situations.

In this paper, we propose a novel approach to generate accurate soft shadows for complex virtual scenes in real time. A Multi-Layer Occluder buffer (MLO-buffer) is used in our algorithm, and it is determined by rendering the scene from the center of area light source. MLO-buffer records all the potential occluders in the scene. When calculating shadow factors, area light source is represented by a set of point samples, and MLO-buffer is traversed to figure out the visibility relations between an occludee and each light sample. Finally, the accurate soft shadows can be produced. The experimental results show that our algorithm can generate high-quality soft shadows in real time without making any assumption, like shape orientation or density distribution, on the area light source.

In this paper, we introduce a novel method to accelerate Screen Space Ambient Occlusion(SSAO) by exploiting spatial and temporal coherence. SSAO computation only depends on the surrounding pixels which is modeled as screen space local complexity function in this paper. Two pixels with similar local complexity should have close Ambient Occlusion(AO) value. Based on the above observation, the AO value of pixel p in the current frame might be directly adopted from some pixel p' in previous frame. The experimental results indicate that our algorithm can improve the performance of SSAO by 20-40 percent but achieve the similar visual effect.

In this paper, a novel algorithm named Distorted Shadow Maps (DSMs) is proposed to generate high-quality hard shadows in real-time. The method focuses on addressing the shadow aliasing caused by different sample distribution between light and camera space. Inspired by the fact that such aliasing occurs in the depth-discontinuous regions of shadow map, in DSMs, a sample redistribution mechanism is designed to enlarge the geometric shadow silhouette regions by shrinking the regions that are completely in light or in shadows. Consequently, more texels in the shadow map are covered by the geometric silhouettes, indicating that silhouettes get more samples. The experimental results show that the jagged edges of hard shadows are reduced by the DSMs algorithm.

In this paper, we propose a novel method that represents the highly-complex point sets by clustering the points to normal-mapped B-spline surfaces (NBSs). The main idea is to construct elaborate normal maps on simple surfaces for the realistic rendering of complex point-set models. Based on this observation, we developed the coarse, normal-mapped B-spline surfaces to approximate the original point datasets with fine surface details. In our algorithm, a genetic clustering algorithm is proposed to automatically segment the point samples into several clusters according to their statistical properties, and a network of B-spline patches with normal maps are constructed according to the clustering results. Our experimental results show that this representation facilitates the modeling and rendering of complex point sets without losing the visual qualities.

In this paper we propose a hybrid representation of environment models by a point representation augmented to geometric representation of surface polygons reconstructed from multiple reference images, through which a real time walkthrough of a complex environment can be achieved. By the method, we start from classification of pixels of the source images into two categories, corresponding respectively to the planar and non-planar surfaces in 3D space. For the pixels corresponding to the planar surfaces, the plane coefficients are reconstructed and all their appearances in the reference images are merged to form uniformly sampled texture images by a comparison of sampling rate and resampling. For the pixels corresponding to the non-planar surfaces, a point representation is applied and the redundant pixels are removed again through sampling rate comparison. The remained pixels are organized by OBB-tree according to their space coordinates. At the same time, the holes that are unable to be captured by all the reference images are pre-filled in the preprocessing phase so that the probability of hole appearance in walkthrough is greatly reduced. Under this hybrid representation, texture mapping and point warping are employed to render the novel views, to take full advantages of the acceleration utility of graphics hardware.

In this paper, a hybrid representation of environment models using a combination of points and polygons is proposed. Through the model, a real time walkthrough of a complex environment can be achieved. We start from multiple depth reference images and classify the pixels of images into two categories, corresponding respectively to planar and non-planar surfaces in 3D space, then all the redundant points are removed by a comparison algorithm of sampling rate. For pixels corresponding to nonplanar surfaces, the point representation is maintained, and a local reconstruction and resampling process is employed to re-sample a set of points that distribute more uniformly on the surfaces. The re-sampled points are organized by an OBB-tree according to their space coordinates and a multiresolution structure is built to improve the rendering efficiency. For pixels corresponding to planar surfaces, the plane coefficients are reconstructed and all their appearances in the reference images are merged to texture maps. Under this hybrid representation, texture mapping and point-based rendering are employed to render the novel views, to take full advantage of the acceleration utility of graphics hardware. The algorithm demonstrates the combined advantages of approaches in traditional computer graphics and PBR/IBR, texture mapping for plane surfaces and point-based rendering for high detail surfaces and shapes.