Accurate disparity estimation in diverse and complex scenes remains a significant challenge in stereo matching, requiring precise geometric perception and robust generalization. Traditional methods often struggle in capturing fine-grained details and maintaining structural consistency under varying conditions, leading to a great reduction of the disparity estimation performance. To address these limitations, we propose Reg-Stereo, a novel framework based on region-aware distribution optimization. It leverages region-aware awakening to extract structural cues and explicitly optimizes the spatial distribution of feature responses. By strengthening relative structural attributes within local regions and expanding them to the global context, our approach enables a more precise and context-aware representation of geometric structures, effectively capturing fine details while preserving global consistency. This innovative approach enables the framework to adapt effectively to diverse and challenging environments, improving both robustness and generalization. Extensive experiments on multiple datasets validate the effectiveness of Reg-Stereo, surpassing exisitng state-of-the-art methods in disparity estimation with enhanced adaptability across complex and heterogeneous scenarios.

In this study, we present a novel method for pinpointing landmarks in X-ray images, which simultaneously offers computational efficiency and localization precision. Our method leverages a cyclic coordinate-guided strategy that requires fewer model parameters and lower computational costs than traditional heatmap-based supervised methods. This is crucial for medical imaging applications where imaging devices often have limited computational resources yet require high-precision landmark localization. Our methodology involves a two-stage process that employs cyclic inference to optimize landmark localization. In the first stage, non-uniform sampling is used to capture the multiscale features of landmarks. This is followed by a second stage in which cyclic training fine-tunes the landmark coordinates towards their optimal positions. Our results indicate that our two-stage process achieves competitive localization performance with state-of-the-art methods yet with added benefits of lower computational overhead and smaller parameter count. Additionally, a global block was developed to capture global position information of landmarks, and experiments showed its effectiveness and its contribution in enhancing the model's landmark localization accuracy. We validated our method using two publicly available datasets, and the source code for our experiments is available on GitHub: https://github.com/switch626/CCG-CL.git.

Personalized Federated Learning (PFL) aims to learn a custom model for each distributed client while benefiting from collaborative training in order to overcome the detrimental impact of data heterogeneity. Despite the promising benefits, the existing approaches often compromise the generalization performance of personalized models, as they solely focus on enhancing the personalization capability of models or merely aim to strike a balance between personalization and generalization. Indeed, increasing the personalization capability while preserving the strong generalization performance enabled by collaborative training remains a challenge for PFL, as the two objectives seem to compete with each other. To tackle this challenge, we investigate the relationship between model generalization and personalization under different degrees of heterogeneity. We find that besides the client-specific data distribution, the distribution bias between the unique data distribution of each client and that of the whole population is another critical factor that prominently impacts these two performances. Motivated by the above finding, we propose PFed-DBA, a novel PFL framework that effectively perceives this distribution bias to guide the training process. Concretely, we design the PFL models as a skip-connection network between a shared module for learning the shared representations delivering the common distribution of data across all clients and a personalized module for learning the personalized representations of the heterogeneous distribution bias. Then, we devise corresponding loss functions, aggregation strategy, and updating strategy in order to make the two modules intelligently complement each other. Moreover, we conduct extensive experiments to evaluate the effectiveness of PFed-DBA. The results show that PFed-DBA improves model accuracy to 12.34% at best compared with the state-of-the-art.

In this study, we present WebCeph2k, an extensive and diverse cephalometric landmark localization dataset that surpasses previous benchmark datasets in terms of number of landmark annotations. This diverse cephalometric landmarks dataset has significant value in medical imaging research. Existing studies predominantly focus on datasets obtained from a single medical center and provider, which offers a limited number of landmarks and a limited diversity of cephalograms, resulting in models that exhibit low robustness and generalization when applied to more diverse datasets. The clinical application of cephalometry is hampered by significant localization errors in landmark localization models, in addition to the inadequacy of existing datasets’ landmarks for clinical cephalometric diagnosis. The limited generalization ability and the occurrence of “overfitting” in deep learning models are mainly caused by the small size in the dataset. In the medical field, the inclusion of large and diverse datasets can greatly improve the generalization and performance of landmark localization models. This paper presents our WebCeph2k dataset from 9 medical centers, covering 9 different imaging devices, which surpasses the only publicly available ISBI2015 dataset in terms of sample size and number of landmarks. In addition, this study employs a low computational cost methodology to achieve optimal landmarks localization: 1) ROI regions of X-ray images are derived by exploiting the prior distribution of the data, 2) the model computational cost is reduced by adopting a spatial-depth transformation strategy, 3) the standard heatmap decoding method is optimized by integrating a compensation strategy. The results show that the proposed method not only achieves competitive localization results to other state-of-the-art approaches, but also offers a reduction of the model computational cost, resulting in faster inference. Consequently, this research offers valuable prospects in the field of general-purpose medical landmark localization methods. We also find that our proposed dataset is more complex and challenging than the ISBI dataset. The dataset and code are available at https://github.com/switch626/WebCeph2k.

Recently, the methods that combine the merits of the numerical model and the deep learning to improve the prediction accuracy of the sea surface temperature (SST) have received considerable attention. Existing methods usually apply the output of the numerical model as the physical knowledge to guide the training of the deep learning models. However, the physical knowledge in the observed data has not been fully exploited. With the development of observational instruments and techniques, an increasing amount of observational data has been collected. These data can be utilized for the exploration of physical knowledge. Toward this end, we propose novel scheme for SST prediction, which applies generative adversarial networks (GANs) to analyze the physical knowledge in the historical data. In particular, two GAN models are trained with numerical model data and observed data separately. Afterward, the physical knowledge is extracted from the observed data which is not contained in the data generated by the numerical model by comparing the learned physical feature from the two pre-trained GAN models. Finally, to validate the relevance of the physical knowledge which we have discovered, the extracted features are added into the numerical model data which are called newly corrected data. Besides, we train two spatial-temporal models over the newly corrected dataset and the original numerical model data for SST prediction, respectively. The experimental results show that the newly corrected dataset performs better than using the original numerical model for SST prediction.

Recent advances on feature descriptors for image matching put more emphasis on encoding invariances (e.g. illumination invariance) to promote the descriptors’ discriminative power. However, according to the information entropy, more invariance implies greater certainty and less informativeness in a descriptor. Consequently, descriptors encoding too many invariances usually show poor generalization to unknown image changes, lacking enough informativeness to cover the large uncertainty in unseen scenes. This limits the application scenarios of learned descriptors. In this paper, we propose to alleviate this issue from the perspective of informativeness and we thus design hierarchical consistent constraint by introducing regression feedback in a self-supervised manner. Combined with the hardest-within-batch matching constraint, we form a novel dual supervision framework, to encourage the descriptor to learn an informative representation while maintaining a good discriminative power. Moreover, to fully mine the context information hidden in image and boost the informativeness in turn, we present AANet, a descriptor network that efficiently predicts dense description by the powerful Attentional Aggregation of multi-level features. Experiments across challenging feature matching on HPatches, RDNIM datasets, and visual localization tasks on Aachen Day-night dataset show that our method outperforms recent state-of-the-art descriptors while keeping encouraging efficiency. The application of visual 3D reconstruction on various scenarios also demonstrates the high generalization ability of our method.

Traditionally, numerical models have been deployed in oceanography studies to simulate ocean dynamics by representing physical equations. However, many factors pertaining to ocean dynamics seem to be ill-defined. We argue that transferring physical knowledge from observed data could further improve the accuracy of numerical models when predicting Sea Surface Temperature (SST). Recently, the advances in earth observation technologies have yielded a monumental growth of data. Consequently, it is imperative to explore ways in which to improve and supplement numerical models utilizing the ever-increasing amounts of historical observational data. To this end, we introduce a method for SST prediction that transfers physical knowledge from historical observations to numerical models. Specifically, we use a combination of an encoder and a generative adversarial network (GAN) to capture physical knowledge from the observed data. The numerical model data is then fed into the pre-trained model to generate physics-enhanced data, which can then be used for SST prediction. Experimental results demonstrate that the proposed method considerably enhances SST prediction performance when compared to several state-of-the-art baselines.

Sea subsurface temperature, an essential component of aquatic wildlife, underwater dynamics, and heat transfer with the sea surface, is affected by global warming in climate change. Existing research is commonly based on either physics-based numerical models or data-based models. Physical modeling and machine learning are traditionally considered as two unrelated fields for the sea subsurface temperature prediction task, with very different scientific paradigms (physics-driven and data-driven). However, we believe that both methods are complementary to each other. Physical modeling methods can offer the potential for extrapolation beyond observational conditions, while data-driven methods are flexible in adapting to data and are capable of detecting unexpected patterns. The combination of both approaches is very attractive and offers potential performance improvement. In this article, we propose a novel framework based on a generative adversarial network (GAN) combined with a numerical model to predict sea subsurface temperature. First, a GAN-based model is used to learn the simplified physics between the surface temperature and the target subsurface temperature in the numerical model. Then, observation data are used to calibrate the GAN-based model parameters to obtain a better prediction. We evaluate the proposed framework by predicting daily sea subsurface temperature in the South China Sea. Extensive experiments demonstrate the effectiveness of the proposed framework compared to existing state-of-the-art methods.

The cold-start problem, faced when providing recommendations to newly joined users with no historical interaction record existing in the platform, is one of the most critical problems that negatively impact the performance of a recommendation system. Fortunately, cross-domain recommendation~(CDR) is a promising approach for solving this problem, which can exploit the knowledge of these users from source domains to provide recommendations in the target domain. However, this method requires that the central server has the interaction behaviour data in both domains of all the users, which prevents users from participating due to privacy issues.

Textures contain a wealth of image information and are widely used in various fields such as computer graphics and computer vision. With the development of machine learning, the texture synthesis and generation have been greatly improved. As a very common element in everyday life, wallpapers contain a wealth of texture information, making it difficult to annotate with a simple single label. Moreover, wallpaper designers spend significant time to create different styles of wallpaper. For this purpose, this paper proposes to describe wallpaper texture images by using multi-label semantics. Based on these labels and generative adversarial networks, we present a framework for perception driven wallpaper texture generation and style transfer. In this framework, a perceptual model is trained to recognize whether the wallpapers produced by the generator network are sufficiently realistic and have the attribute designated by given perceptual description; these multi-label semantic attributes are treated as condition variables to generate wallpaper images. The generated wallpaper images can be converted to those with well-known artist styles using CycleGAN. Finally, using the aesthetic evaluation method, the generated wallpaper images are quantitatively measured. The experimental results demonstrate that the proposed method can generate wallpaper textures conforming to human aesthetics and have artistic characteristics.

3D hand pose estimation from a monocular RGB image is a highly challenging task due to self-occlusion, diverse appearances, and inherent depth ambiguities within monocular images. Most of the previous methods first employ deep neural networks to fit 2D joint location maps, then combines them with implicit or explicit pose-aware features to directly regress 3D hand joints positions using their designed network structure. However, the skeleton positions and corresponding skeleton-aware content information located in the latent space are invariably ignored. These skeleton-aware contents effectively bridge the gap between hand joint and hand skeleton information by associating the relationship between different hand joints features and the hand skeleton positions distribution in 2D space. To address this issue, we propose a simple yet efficient deep neural network to directly recover reliable 3D hand pose from monocular RGB images with faster estimation process. Our purpose is the reduction of the model computational complexity while maintaining high precision performance. Therefore, we design a novel Feature Chat Block (FCB) to complete feature boosting, which enables the intuitively enhanced interaction between joint and skeleton features. First, this FCB module updates joint features effectively based on semantic graph convolutional neural network and multi-head self-attention mechanism. The GCN-based structure focuses on the physical hand joints included in a binary adjacency matrix and the self-attention part pays attention to hand joints located in a complementary matrix. Then, the FCB module employs query and key mechanisms respectively representing joint and skeleton features to further implement feature interaction. After a set of FCB modules, our model updates the fused features in a coarse-to-fine manner and finally outputs a predicted 3D hand pose. We conducted a comprehensive set of ablation experiments on the InterHand2.6M dataset to validate the effectiveness and significance of the proposed method. Additionally, experimental results on Rendered Hand Dataset, Stereo Hand Datasets, First-Person Hand Action Dataset and FreiHAND Dataset show our model surpasses the state-of-the-art methods with faster inference speed.

In recent years, marine animal study has gained increasing research attention, which raises significant demands for fine-grained marine animal segmentation (MAS) techniques. In addition, deep learning has been widely adopted for object segmentation and has achieved promising performance. However, deep-based MAS is still lack of investigation due to the shortage of a large-scale MAS dataset. To tackle this issue, we construct the first large-scale MAS dataset, calledMAS3K, which consists of 3,103 images from different types, including camouflaged marine animal images, common marine animal images, and underwater images without marine animals. Furthermore, we consider different underwater conditions, such as low illumination, turbid water quality, photographic distortion, etc. Each image fromMAS3Kdataset has rich annotations, including an object-level mask, a category name, attributes, and a camouflage method (if applicable). Furthermore, we propose a novel MAS network, called Enhanced Cascade Decoder Network (ECD-Net), which consists of multiple Interactive Feature Enhancement Modules (IFEMs) and Cascade Decoder Modules (CDMs). InECD-Net, the IFEMs are first utilized to extract rich multi-scale features. The resulting features are then fed to the CDMs for accurately segmenting marine animals from complex underwater environments. We perform extensive experiments to compareECD-Netwith 10 cutting-edge object segmentation models. The results demonstrate thatECD-Netis an effective MAS model and outperforms the cutting-edge models, both qualitatively and quantitatively.

Hand pose estimation in 3D space from a single RGB image is a highly challenging problem due to self-geometric ambiguities, diverse texture, viewpoints, and self-occlusions. Existing work proves that a network structure with multi-scale resolution subnets, fused in parallel can more effectively shows the spatial accuracy of 2D pose estimation. Nevertheless, the features extracted by traditional convolutional neural networks cannot efficiently express the unique topological structure of hand key points based on discrete and correlated properties. Some applications of hand pose estimation based on traditional convolutional neural networks have demonstrated that the structural similarity between the graph and hand key points can improve the accuracy of the 3D hand pose regression. In this paper, we design and implement an end-to-end network for predicting 3D hand pose from a single RGB image. We first extract multiple feature maps from different resolutions and make parallel feature fusion, and then model a graph-based convolutional neural network module to predict the initial 3D hand key points. Next, we use 2D spatial relationships and 3D geometric knowledge to build a self-supervised module to eliminate domain gaps between 2D and 3D space. Finally, the final 3D hand pose is calculated by averaging the 3D hand poses from the GCN output and the self-supervised module output. We evaluate the proposed method on two challenging benchmark datasets for 3D hand pose estimation. Experimental results show the effectiveness of our proposed method that achieves state-of-the-art performance on the benchmark datasets.