Deep learning has been widely applied in various domains. Current widely-used optimizers, such as SGD, Adam, and their variants, are designed based on the assumption that the gradient noise generated during model training follows a Gaussian distribution. However, recent empirical studies have found that the gradient noise often does not follow a Gaussian distribution. Instead, the noise exhibits heavy-tailed characteristics consistent with an $\alpha $ -stable distribution, casting doubt on the performance and robustness of optimizers designed under the assumption of Gaussian noise. Inspired by the least mean p-power (LMP) algorithm from the field of adaptive filtering, we propose a novel optimizer called Ape for deep learning. Ape integrates a p-power adjustment mechanism to compress large gradients and amplify small ones, mitigating the impact of heavy-tailed gradient distributions. It also employs an approach for estimating second moments tailored to $\alpha $ -stable distributions. Extensive experiments on benchmark datasets demonstrate Ape's effectiveness in improving both accuracy and training speed compared to existing optimizers. The Ape optimizer showcases the potential of cross-disciplinary approaches in advancing deep learning optimization techniques and lays the groundwork for future innovations in this domain.

Learning from large heterogeneous graphs presents significant challenges due to the scale of networks, heterogeneity in node and edge types, variations in nodal features, and complex local neighborhood structures. This paper advocates for ensemble learning as a natural solution to this problem, whereby training multiple graph learners under distinct sampling conditions, the ensemble inherently captures different aspects of graph heterogeneity. Yet, the crux lies in combining these learners to meet global optimization objective while maintaining computational efficiency on large-scale graphs. In response, we propose LHGEL, an ensemble framework that addresses these challenges through batch sampling with three key components, namely batch view aggregation, residual attention, and diversity regularization. Specifically, batch view aggregation samples subgraphs and forms multiple graph views, while residual attention adaptively weights the contributions of these views to guide node embeddings toward informative subgraphs, thereby improving the accuracy of base learners. Diversity regularization encourages representational disparity across embedding matrices derived from different views, promoting model diversity and ensemble robustness. Our theoretical study demonstrates that residual attention mitigates gradient vanishing issues commonly faced in ensemble learning. Empirical results on five real heterogeneous networks validate that our LHGEL approach consistently outperforms its state-of-the-art competitors by substantial margin. Codes and datasets are available at https://github.com/Chrisshen12/LHGEL.

Quantifying sperm flagellar beating behavior (e.g., beating amplitude, frequency, and wavelength) plays a crucial role in biological research, clinical diagnostics, and the design of sperm-inspired microrobots. However, existing computational methods struggle to accurately and efficiently analyze the highly dynamic, complex, and fine structures of sperm flagella, especially when portions of the flagellum become invisible due to three-dimensional out-of-focus beating. This paper proposes an automated high-throughput tool for quantitative analysis of sperm flagellar beating. The core innovation is continuous convolution (CConv), which adaptively captures the irregular, time-varying patterns of sperm flagella while ensuring continuity in segmentation outputs, even in the presence of locally invisible regions caused by out-of-focus motion. CConv can be integrated into various neural network architectures as a plug-and-play module. Extensive experiments demonstrate that integrating CConv consistently improves the accuracy and continuity of flagella segmentation across different networks. Furthermore, utilizing a curvature-based approach, we quantified key flagellar beating parameters, including length, amplitude, frequency, and wavelength. Applying the high-throughput tool on 1200 sperm revealed that sperm from fertile donors had significantly higher flagellar beating frequency than sperm from infertile patients. The proposed automated tool unlocks high-throughput, quantitative analysis of sperm flagellar beating, showing the potential for applications in reproductive biology and engineering research. The codes and datasets will be released at https://github.com/Goldfish-Yu/CConv.

This paper presents HGEN that pioneers ensemble learning for heterogeneous graphs. We argue that the heterogeneity in node types, nodal features, and local neighborhood topology poses significant challenges for ensemble learning, particularly in accommodating diverse graph learners. Our HGEN framework ensembles multiple learners through a meta-path and transformation-based optimization pipeline to uplift classification accuracy. Specifically, HGEN uses meta-path combined with random dropping to create Allele Graph Neural Networks (GNNs), whereby the base graph learners are trained and aligned for later ensembling. To ensure effective ensemble learning, HGEN presents two key components:1) a residual-attention mechanism to calibrate allele GNNs of different meta-paths, thereby enforcing node embeddings to focus on more informative graphs to improve base learner accuracy, and 2) a correlation-regularization term to enlarge the disparity among embedding matrices generated from different meta-paths, thereby enriching base learner diversity. We analyze the convergence of HGEN and attest its higher regularization magnitude over simple voting. Experiments on five heterogeneous networks validate that HGEN consistently outperforms its state-of-the-art competitors by substantial margin. Codes are available at https://github.com/Chrisshen12/HGEN.

Graph Neural Networks (GNNs) have shown superb performance in handling networked data, mainly attributed to their message passing and convolution process across neighbors. For most literature, the performance of GNNs is mainly reported based on noise-free data environments. No study has systematically evaluated GNNs’ performance under noise. In this article, we carry out an empirical study and theoretical analysis of four types of GNNs, including Graph Convolutional Networks (GCNs), Graph Attention Networks (GATs), Graph Contrastive Networks (GCL), and graph UniFilter under three types of noise, including attribute noise, structure noise, and label noise. Our study shows that GNNs behave tremendously differently in response to different types of noise. Overall, GAT is the most noise vulnerable and sensitive, whereas GCL is the most noise resilient. We further carry out theoretical analysis to explain the reason causing GAT to be sensitive to noise, and propose a solution to enhance its noise resilience. Our study brings in-depth firsthand knowledge of GNNs under noise for researchers and practitioners to better utilize GNNs in real-world applications.

Label Distribution Learning (LDL), as a more general learning setting than generic single-label and multi-label learning, has been commonly used in computer vision and many other applications. To date, existing LDL approaches are designed and applied to data without considering the interdependence between instances. In this paper, we propose a Graph Label Distribution Learning (GLDL) framework, which explicitly models three types of relationships: instance-instance, label-label, and instance-label, to learn the label distribution for networked data. A label-label network is learned to capture label-to-label correlation, through which GLDL can accurately learn label distributions for nodes. Dual graph convolution network (GCN) Co-training with heterogeneous message passing ensures two GCNs, one focusing on instance-instance relationship and the other one targeting label-label correlation, are jointly trained such that instance-instance relationship can help induce label-label correlation and vice versa. Our theoretical study derives the error bound of GLDL. For verification, four benchmark datasets with label distributions for nodes are created using common graph benchmarks. The experiments show that considering dependency helps learn better label distributions for networked data, compared to state-of-the-art LDL baseline. In addition, GLDL not only outperforms simple GCN and graph attention networks (GAT) using distribution loss but is also superior to its variant considering label-label relationship as a static network. GLDL and its benchmarks are the first research endeavors to address LDL for graphs. Code and benchmark data are released for public access.

Graph neural networks (GNN) have been commonly used for learning and classifying objects with correlated relationships. To date, many GNN architectures exist, but majority of them only work well on shallow networks due to the oversmoothing phenomenon, where node features become similar to each other, as the layer increases. In this paper, we point out that the key to create an informative deep GNN is to have an adaptive feature updating rate control for each node, where the updating rate should take each node's locality into consideration through shared trainable weight parameters. Accordingly, we advocate a new graph rhythm modeling as a generalized mechanism to the Dirichlet energy based approaches. Instead of merely modeling difference between nodes, like Dirichlet energy based approach does, graph rhythm focuses on omni-directional relationship mapping between each node and its neighbors. Such a mechanism provides a more general ways of capturing patterns between nodes (i.e. graph rhythm) for effective graph neural network learning. Experiments and comparisons, demonstrate the performance gain and show that GRN can help create GNNs with deep layers, without suffering from performance deterioration or having better performance than shallow networks.

Obtaining three-dimensional information, especially the z-axis depth information, is crucial for robotic micromanipulation. Due to the unavailability of depth sensors such as lidars in micromanipulation setups, traditional depth acquisition methods such as depth from focus or depth from defocus directly infer depth from microscopic images and suffer from poor resolution. Alternatively, micromanipulation tasks obtain accurate depth information by detecting the contact between an end-effector and an object (e.g., a cell). Despite its high accuracy, only sparse depth data can be obtained due to its low efficiency. This paper aims to address the challenge of acquiring dense depth information during robotic cell micromanipulation. A weakly-supervised depth completion network is proposed to take cell images and sparse depth data obtained by contact detection as input to generate a dense depth map. A two-stage data augmentation method is proposed to augment the sparse depth data, and the depth map is optimized by a network refinement method. The experimental results show that the MAE value of the depth prediction error is less than 0.3 µm, which proves the accuracy and effectiveness of the method. This deep learning network pipeline can be seamlessly integrated with the robotic micromanipulation tasks to provide accurate depth information.

Label Distribution Learning (LDL) has been extensively studied in IID data applications such as computer vision, thanks to its more generic setting over single-label and multi-label classification. This paper advances LDL into graph domains and aims to tackle a novel and fundamental heterogeneous graph label distribution learning (HGDL) problem. We argue that the graph heterogeneity reflected on node types, node attributes, and neighborhood structures can impose significant challenges for generalizing LDL onto graphs. To address the challenges, we propose a new learning framework with two key components: 1) proactive graph topology homogenization, and 2) topology and content consistency-aware graph transformer. Specifically, the former learns optimal information aggregation between meta-paths, so that the node heterogeneity can be proactively addressed prior to the succeeding embedding learning; the latter leverages an attention mechanism to learn consistency between meta-path and node attributes, allowing network topology and nodal attributes to be equally emphasized during the label distribution learning. By using KL-divergence and additional constraints, \method~delivers an end-to-end solution for learning and predicting label distribution for nodes. Both theoretical and empirical studies substantiate the effectiveness of our HGDL approach. Our code and datasets are available at https://github.com/Listener-Watcher/HGDL.

The agile and safe transdomain in waves is a promising feature but the primary bottleneck of the hybrid aerial underwater vehicle (HAUV). In this article, the surfing algorithm is proposed for Nezha-mini, our predeveloped HAUV prototype, to search for the dynamic window facilitating takeoff in waves and avoiding hazardous waves. For the first time, the cross-domain window, i.e., the vehicle is at the wave crest and heading downstream, is characterized and defined through the vehicle-wave coupled dynamic model. The novel surfing algorithm consists of the gradient perceptron, time-limited momentum gradient search, heading server, and initial conditions. Nezha-mini senses, searches, and tracks the dynamic window in real-time, until the takeoff decisions are triggered. Numerical simulations and experiments in regular and irregular waves reveal the effectiveness of the algorithm. The vehicle maintains a healthy initial attitude and inaccessible wave disturbance during takeoff, thus alleviating the thrust distraction from stability recovery and uncertainty. The average transition time and energy cost are reduced by 59.2% and 26.1% compared with random takeoff cases, and the locomotion is smooth, graceful, and low-risk. The computation and cost are low as the algorithm only requires the basic flight controller and the data from the inertial measurement unit instead of the prior parameters of the HAUV and waves. In comparison with the adaptive robust controller, which resists wave disturbance directly, this article provides an enlightening strategy from the perspective of harnessing waves.

Graph Contrastive Learning (GCL) has recently emerged to leverage contrastive loss as a pseudo-supervision signal for self-supervised learning. In order to introduce contrastive learning loss to graphs, existing GCL methods mostly focus on leveraging network topology or node similarity to classify a pair of nodes as same/different node pairs or close/distant node pairs. In this paper, we propose a semi-supervised graph contrastive learning framework, pmGCL, leveraging GCL to augment the performance of a classifier through a predictive masking approach. Specifically, a classifier is trained using a small number of labeled nodes to predict node labels. The label prediction results are then transformed into a binary prediction of whether two nodes have the same label or not for all node pairs. The converted result, serving as a binary masking matrix, will help the succeeding GCL learning to learn to pull nodes likely belonging to the same class to be closer and push the ones belonging to different classes to be further away from each other. Experiments and comparisons, with respect to different benchmark networks and label percentages, show that pmGCL consistently outperforms rival graph convolution neural network (GCN) and GCL baseline with a simple constraint posed on the problem.

This paper proposes an accelerated proximal stochastic variance reduced gradient (ASVRG) method, in which we design a simple and effective momentum acceleration trick. Unlike most existing accelerated stochastic variance reduction methods such as Katyusha, ASVRG has only one additional variable and one momentum parameter. Thus, ASVRG is much simpler than those methods, and has much lower per-iteration complexity. We prove that ASVRG achieves the best known oracle complexities for both strongly convex and non-strongly convex objectives. In addition, we extend ASVRG to mini-batch and non-smooth settings. We also empirically verify our theoretical results and show that the performance of ASVRG is comparable with, and sometimes even better than that of the state-of-the-art stochastic methods.