Existing multi-organ segmentation methods usually rely on large and fully labeled datasets for training. However, medical image datasets are typically decentralized by privacy constraints and partially labeled due to the high costs of full annotation in clinical practice, resulting in label inconsistency across medical centers. Federated learning offers privacy-preserving decentralized training, but the label inconsistency leads to significant divergence in local model parameters across medical centers, thereby hindering the achievement of the global optimum. To resolve this issue, an effective and communication-efficient Federated Learning under Reliable Supervision (FedRS) is proposed, which ensures: i) the local models are trained with reliable supervisory information through the proposed Less-Forgetting and Less-Constraint loss functions, thereby reducing the divergence in local model parameters; and ii) the global model is aggregated based on the consistency of predictions between each local model (after local training) and the global model (received before training), thereby enhancing the reliability of the global model. Extensive experimental results on nine publicly available 3D abdominal CT image datasets show that our FedRS outperforms localized, centralized, and state-of-the-art federated learning methods on both in-federation and out-of-federation datasets, demonstrating its effectiveness and strong generalization capability. In particular, our FedRS only utilizes a model with only 4.1M parameters as its backbone, thereby significantly reducing its communication cost. The source code is publicly available at https://github.com/luohy812/FedRS.

Retinal image segmentation is essential for analyzing retinal structures like vessels and diagnosing retinopathy. However, the inherent intricacy of the retina, along with annotation scarcity and data heterogeneity, presents prevalent challenges in creating accurate and generalizable deep learning models. Diffusion models, while initially developed for image generation, have recently shown great promise for visual perception by leveraging the learned internal representations. However, these diffusion representations, which spread across network blocks (space) and diffusion timesteps (time), potentially suffer from issues like stochastic semantic distortion and cumulative structural blurring, compromising their semantic fidelity to the source image. In this paper, by delving into the generalization property of diffusion models, we propose a novel anchoring inversion strategy to derive diffusion representations that are semantically faithful to the source image from the deterministic trajectory. Furthermore, we introduce a time-space frequency-aware aggregation interpreter (T&S-FreqAgg) to aggregate the multi-scale and multi-timestep diffusion representations in a frequency-aware way for Domain Generalizable Semantic Segmentation (DGSS). Extensive experiments on nine public retinal image datasets demonstrate the superiority of our proposed framework, DiffDGSSv2, over state-of-the-art methods. Our code will be available at: https://github.com/Xyporz/DiffDGSSv2.

Federated Class Incremental Learning (FCIL) enables distributed clients to collaboratively train a global model based on their private sequential tasks without compromising data privacy. Currently, some FCIL methods have been proposed, and most are designed based on deep models. However, enabling these FCIL models to converge requires numerous communication rounds, significantly increasing communication costs. Recently, the Broad Learning System (BLS), an effective and efficient shallow model, was proposed and adapted for CIL tasks [i.e., BLS-Class Incremental Learning (CIL)]. BLS-CIL exhibits fast updates and high retainability. However, it requires prior knowledge of when new class data arrives and cannot be directly used in federated scenarios due to the global catastrophic forgetting in FCIL. Thus, an innovative Federated Class incremental learning method based on BLS (FedCBLS) is proposed, which extends BLS-CIL within the federated scenario and provides three advantages: 1) high accuracy from the local perspective, achieved by integrating BLS-CIL with a newly designed automatic decision-making (ADM) method to detect novel classes and learn them incrementally for local clients; 2) high accuracy from the global perspective, attained through the newly proposed local model refinement (LMR) and global model projection (GMP) methods, mitigating global catastrophic forgetting stemming from heterogeneous data across clients; and 3) extremely low communication costs due to the newly derived closed-form solutions without iterative optimization for both local and global models. Comprehensive experimental results show that our FedCBLS outperforms the state-of-the-art (SOTA) FCIL methods by up to 8.15%, while drastically reducing communication costs to 1% of SOTA’s. Our code is available athttps://github.com/dujie-szu/FedCBLS.git

Visual place recognition (VPR) has played an essential role in simultaneous localization and mapping-based mobile robotics and autonomous driving in the past decade, which can identify previously visited places by matching the current observed view against a view database for global localization and loop closure. However, existing VPR methods always suffer fromfalse place recognitiondue to the following issues: insensitive spatial–temporal embedding extraction, lack of multiview descriptor for matching, and misconsideration of false views for model optimization. To address these issues, a novel robust framework calledVPR under variational views (VPR-VV)is proposed. VPR-VV is integrated with: a sequence encoder to extract robust spatial–temporal features from a view sequence, then a hierarchical view retrieval module is employed for multiview feature descriptor aggregation, and a novel enhanced ranking feedback average precision loss with the normalized discounted cumulative gain metric is designed for model optimization. As a result, VPR-VV can significantly enhance the accuracy and robustness of VPR for robot localization under variational views. Experiments with ablation studies are conducted on various challenging indoor and outdoor datasets, and our framework’s superiority is demonstrated: VPR-VV outperforms state-of-the-art (SOTA) methods by up to 9.4% in recall@1, and real-time inference is achieved without additional memory or computational overhead.

Class activation mapping (CAM) methods have garnered considerable research attention because they can be used to interpret the decision-making of deep convolutional neural network (CNN) models and provide initial masks for weakly supervised semantic segmentation (WSSS) tasks. However, the class activation maps generated by most CAM methods usually have two limitations: 1) a lack of the ability to cover the whole object when using low-level features; and 2) introducing background noise. To mitigate these issues, an innovative Context-level weights-based CAM (Context-CAM) method is proposed, which guarantees: 1) the non-discriminative regions that have similar appearances and are located close to the discriminative regions can also be highlighted by the newly designed Region-Enhanced Mapping (REM) module using context-level weights; and 2) the background noises are gradually eliminated via a newly proposed Semantic-guided Reverse Sequence Fusion (SRSF) strategy that can sequentially denoise and fuse the region-enhanced maps from the last layer to the first layer. Extensive experimental results show that our Context-CAM can generate higher-quality class activation maps than classic and state-of-the-art (SOTA) CAM methods in terms of the Energy-Based Pointing Game (EBPG) score, and the improvements are up to 35.49% when compared to the second-best method. Moreover, for WSSS tasks, our Context-CAM can directly replace the CAM method used in existing WSSS methods without any architectural modification to further improve the segmentation performance. Our code is available at https://github.com/cwb0611/Context-CAM.

Camouflaged object detection (COD) aims to segment targeted objects that have similar colors, textures, or shapes to their background environment. Due to the limited ability in distinguishing highly similar patterns, existing COD methods usually produce inaccurate predictions, especially around the boundary areas, when coping with complex scenes. This paper proposes a Progressive Region-to-Boundary Exploration Network (PRBE-Net) to accurately detect camouflaged objects. PRBE-Net follows an encoder-decoder framework and includes three key modules. Specifically, firstly, both high-level and low-level features of the encoder are integrated by a region and boundary exploration module to explore their complementary information for extracting the object's coarse region and fine boundary cues simultaneously. Secondly, taking the region cues as the guidance information, a Region Enhancement (RE) module is used to adaptively localize and enhance the region information at each layer of the encoder. Subsequently, considering that camouflaged objects usually have blurry boundaries, a Boundary Refinement (BR) decoder is used after the RE module to better detect the boundary areas with the assistance of boundary cues. Through top-down deep supervision, PRBE-Net can progressively refine the prediction. Extensive experiments on four datasets indicate that our PRBE-Net achieves superior results over 21 state-of-the-art COD methods. Additionally, it also shows good results on polyp segmentation, a COD-related task in the medical field.

Machine learning aims to generate a predictive model from a training dataset of a fixed number of known classes. However, many real-world applications (such as health monitoring and elderly care) are data streams in which new data arrive continually in a short time. Such new data may even belong to previously unknown classes. Hence, class-incremental learning (CIL) is necessary, which incrementally and rapidly updates an existing model with the data of new classes while retaining the existing knowledge of old classes. However, most current CIL methods are designed based on deep models that require a computationally expensive training and update process. In addition, deep learning based CIL (DCIL) methods typically employ stochastic gradient descent (SGD) as an optimizer that forgets the old knowledge to a certain extent. In this article, a broad learning system-based CIL (BLS-CIL) method with fast update and high retainability of old class knowledge is proposed. Traditional BLS is a fast and effective shallow neural network, but it does not work well on CIL tasks. However, our proposed BLS-CIL can overcome these issues and provide the following: 1) high accuracy due to our novel class-correlation loss function that considers the correlations between old and new classes; 2) significantly short training/update time due to the newly derived closed-form solution for our class-correlation loss without iterative optimization; and 3) high retainability of old class knowledge due to our newly derived recursive update rule for CIL (RULL) that does not replay the exemplars of all old classes, as contrasted to the exemplars-replaying methods with the SGD optimizer. The proposed BLS-CIL has been evaluated over 12 real-world datasets, including seven tabular/numerical datasets and six image datasets, and the compared methods include one shallow network and seven classical or state-of-the-art DCIL methods. Experimental results show that our BIL-CIL can significantly improve the classification performance over a shallow network by a large margin (8.80%-48.42%). It also achieves comparable or even higher accuracy than DCIL methods, but greatly reduces the training time from hours to minutes and the update time from minutes to seconds.

Deep metric learning (DML) has been widely applied in various tasks (e.g., medical diagnosis and face recognition) due to the effective extraction of discriminant features via reducing data overlapping. However, in practice, these tasks also easily suffer from two class-imbalance learning (CIL) problems: data scarcity and data density, causing misclassification. Existing DML losses rarely consider these two issues, while CIL losses cannot reduce data overlapping and data density. In fact, it is a great challenge for a loss function to mitigate the impact of these three issues simultaneously, which is the objective of our proposed intraclass diversity and interclass distillation (IDID) loss with adaptive weight in this article. IDID-loss generates diverse features within classes regardless of the class sample size (to alleviate the issues of data scarcity and data density) and simultaneously preserves the semantic correlations between classes using learnable similarity when pushing different classes away from each other (to reduce overlapping). In summary, our IDID-loss provides three advantages: 1) it can simultaneously mitigate all the three issues while DML and CIL losses cannot; 2) it generates more diverse and discriminant feature representations with higher generalization ability, compared with DML losses; and 3) it provides a larger improvement on the classes of data scarcity and density with a smaller sacrifice on easy class accuracy, compared with CIL losses. Experimental results on seven public real-world datasets show that our IDID-loss achieves the best performances in terms of G-mean, F1-score, and accuracy when compared with both state-of-the-art (SOTA) DML and CIL losses. In addition, it gets rid of the time-consuming fine-tuning process over the hyperparameters of loss function.

In computer-aided diagnosis and treatment planning, accurate segmentation of medical images plays an essential role, especially for some hard regions including boundaries, small objects and background interference. However, existing segmentation loss functions including distribution-, region- and boundary-based losses cannot achieve satisfactory performances on these hard regions. In this paper, a boundary-sensitive loss function with location constraint is proposed for hard region segmentation in medical images, which provides three advantages: i) our Boundary-Sensitive loss (BS-loss) can automatically pay more attention to the hard-to-segment boundaries (e.g., thin structures and blurred boundaries), thus obtaining finer object boundaries; ii) BS-loss also can adjust its attention to small objects during training to segment them more accurately; and iii) our location constraint can alleviate the negative impact of the background interference, through the distribution matching of pixels between prediction and Ground Truth (GT) along each axis. By resorting to the proposed BS-loss and location constraint, the hard regions in both foreground and background are considered. Experimental results on three public datasets demonstrate the superiority of our method. Specifically, compared to the second-best method tested in this study, our method improves performance on hard regions in terms of Dice similarity coefficient (DSC) and 95% Hausdorff distance (95%HD) of up to 4.17% and 73% respectively. In addition, it also achieves the best overall segmentation performance. Hence, we can conclude that our method can accurately segment these hard regions and improve the overall segmentation performance in medical images.

Pixel-level annotations are extremely expensive for medical image segmentation tasks as both expertise and time are needed to generate accurate annotations. Semi-supervised learning (SSL) for medical image segmentation has recently attracted growing attention because it can alleviate the exhausting manual annotations for clinicians by leveraging unlabeled data. However, most of the existing SSL methods do not take pixel-level information (e.g., pixel-level features) of labeled data into account, i.e., the labeled data are underutilized. Hence, in this work, an innovative Coarse-Refined Network with pixel-wise Intra-patch ranked loss and patch-wise Inter-patch ranked loss (CRII-Net) is proposed. It provides three advantages: i) it can produce stable targets for unlabeled data, as a simple yet effective coarse-refined consistency constraint is designed; ii) it is very effective for the extreme case where very scarce labeled data are available, as the pixel-level and patch-level features are extracted by our CRII-Net; and iii) it can output fine-grained segmentation results for hard regions (e.g., blurred object boundaries and low-contrast lesions), as the proposed Intra-Patch Ranked Loss (Intra-PRL) focuses on object boundaries and Inter-Patch Ranked loss (Inter-PRL) mitigates the adverse impact of low-contrast lesions. Experimental results on two common SSL tasks for medical image segmentation demonstrate the superiority of our CRII-Net. Specifically, when there are only 4% labeled data, our CRII-Net improves the Dice similarity coefficient (DSC) score by at least 7.49% when compared to five classical or state-of-the-art (SOTA) SSL methods. For hard samples/regions, our CRII-Net also significantly outperforms other compared methods in both quantitative and visualization results.

Current state-of-the-art class-imbalanced loss functions for deep models require exhaustive tuning on hyperparameters for high model performance, resulting in low training efficiency and impracticality for nonexpert users. To tackle this issue, a parameter-free loss (PF-loss) function is proposed, which works for both binary and multiclass-imbalanced deep learning for image classification tasks. PF-loss provides three advantages: 1) training time is significantly reduced due to NO tuning on hyperparameter(s); 2) it dynamically pays more attention on minority classes (rather than outliers compared to the existing loss functions) with NO hyperparameters in the loss function; and 3) higher accuracy can be achieved since it adapts to the changes of data distribution in each mini-batch instead of the fixed hyperparameters in the existing methods during training, especially when the data are highly skewed. Experimental results on some classical image datasets with different imbalance ratios (IR, up to 200) show that PF-loss reduces the training time down to 1/148 of that spent by compared state-of-the-art losses and simultaneously achieves comparable or even higher accuracy in terms of both G-mean and area under receiver operating characteristic (ROC) curve (AUC) metrics, especially when the data are highly skewed.

Predicting the motion trajectories of moving agents in complex traffic scenes, such as crossroads and roundabouts, plays an important role in cooperative intelligent transportation systems. Nevertheless, accurately forecasting the motion behavior in a dynamic scenario is challenging due to the complex cooperative interactions between moving agents. Graph Convolutional Neural Network has recently been employed to deal with the cooperative interactions between agents. Despite the promising performance of resulting trajectory prediction algorithms, many existing graph-based approaches model interactions with an undirected graph, where the strength of influence between agents is assumed to be symmetric. However, such an assumption often does not hold in reality. For example, in pedestrian or vehicle interaction modeling, the moving behavior of a pedestrian or vehicle is highly affected by the ones ahead, while the ones ahead usually pay less attention to the ones behind. To fully exploit the asymmetric attributes of the cooperative interactions in intelligent transportation systems, in this work, we present a directed graph convolutional neural network for multiple agents trajectory prediction. First, we propose three directed graph topologies, i.e., view graph, direction graph, and rate graph, by encoding different prior knowledge of a cooperative scenario, which endows the capability of our framework to effectively characterize the asymmetric influence between agents. Then, a fusion mechanism is devised to jointly exploit the asymmetric mutual relationships embedded in constructed graphs. Furthermore, a loss function based on Cauchy distribution is designed to generate multimodal trajectories. Experimental results on complex traffic scenes demonstrate the superior performance of our proposed model when compared with existing approaches.

High accuracy of text classification can be achieved through simultaneous learning of multiple information, such as sequence information and word importance. In this article, a kind of flat neural networks called the broad learning system (BLS) is employed to derive two novel learning methods for text classification, including recurrent BLS (R-BLS) and long short-term memory (LSTM)-like architecture: gated BLS (G-BLS). The proposed two methods possess three advantages: 1) higher accuracy due to the simultaneous learning of multiple information, even compared to deep LSTM that extracts deeper but single information only; 2) significantly faster training time due to the noniterative learning in BLS, compared to LSTM; and 3) easy integration with other discriminant information for further improvement. The proposed methods have been evaluated over 13 real-world datasets from various types of text classification. From the experimental results, the proposed methods achieve higher accuracies than LSTM while taking significantly less training time on most evaluated datasets, especially when the LSTM is in deep architecture. Compared to R-BLS, G-BLS has an extra forget gate to control the flow of information (similar to LSTM) to further improve the accuracy on text classification so that G-BLS is more effective while R-BLS is more efficient.

Automatic gastric tumor segmentation and lymph node (LN) classification not only can assist radiologists in reading images, but also provide image-guided clinical diagnosis and improve diagnosis accuracy. However, due to the inhomogeneous intensity distribution of gastric tumor and LN in CT scans, the ambiguous/missing boundaries, and highly variable shapes of gastric tumor, it is quite challenging to develop an automatic solution. To comprehensively address these challenges, we propose a novel 3D multi-attention guided multi-task learning network for simultaneous gastric tumor segmentation and LN classification, which makes full use of the complementary information extracted from different dimensions, scales, and tasks. Specifically, we tackle task correlation and heterogeneity with the convolutional neural network consisting of scale-aware attention-guided shared feature learning for refined and universal multi-scale features, and task-aware attention-guided feature learning for task-specific discriminative features. This shared feature learning is equipped with two types of scale-aware attention (visual attention and adaptive spatial attention) and two stage-wise deep supervision paths. The task-aware attention-guided feature learning comprises a segmentation-aware attention module and a classification-aware attention module. The proposed 3D multi-task learning network can balance all tasks by combining segmentation and classification loss functions with weight uncertainty. We evaluate our model on an in-house CT images dataset collected from three medical centers. Experimental results demonstrate that our method outperforms the state-of-the-art algorithms, and obtains promising performance for tumor segmentation and LN classification. Moreover, to explore the generalization for other segmentation tasks, we also extend the proposed network to liver tumor segmentation in CT images of the MICCAI 2017 Liver Tumor Segmentation Challenge. Our implementation is released at https://github.com/infinite-tao/MA-MTLN.

In this paper, a robust online multilabel learning method dealing with dynamically changing multilabel data streams is proposed. The proposed method has three advantages: 1) higher accuracy due to a newly defined objective function based on labels ranking; 2) fast training and update based on a newly derived closed-form (rather than gradient descent based) solution for the new objective function; and 3) high robustness to a newly identified concept drift in multilabel data streams, namely, changes in data distribution with labels (CDDL). The high robustness benefits from two novel works: 1) a new sequential update rule that preserves the labels ranking information learned from all old (but discarded) samples while updating the model only based on new incoming samples and 2) a fixed threshold for label bipartition that is insensitive to any kind of changes in data distribution including CDDL. The proposed method has been evaluated over 13 benchmark datasets from various domains. As shown in the experimental results, the proposed work is highly robust to CDDL in both the sequential model update and multilabel thresholding. Furthermore, the proposed method improves the performance in different evaluation measures, including Hamming loss, F1-measure, Precision, and Recall while taking short training time on most evaluated datasets.

In this paper, a novel learning method called postboosting using extended G-mean (PBG) is proposed for online sequential multiclass imbalance learning (OS-MIL) in neural networks. PBG is effective due to three reasons. 1) Through postadjusting a classification boundary under extended G-mean, the challenging issue of imbalanced class distribution for sequentially arriving multiclass data can be effectively resolved. 2) A newly derived update rule for online sequential learning is proposed, which produces a high G-mean for current model and simultaneously possesses almost the same information of its previous models. 3) A dynamic adjustment mechanism provided by extended G-mean is valid to deal with the unresolved challenging dense-majority problem and two dynamic changing issues, namely, dynamic changing data scarcity (DCDS) and dynamic changing data diversity (DCDD). Compared to other OS-MIL methods, PBG is highly effective on resolving DCDS, while PBG is the only method to resolve dense-majority and DCDD. Furthermore, PBG can directly and effectively handle unscaled data stream. Experiments have been conducted for PBG and two popular OS-MIL methods for neural networks under massive binary and multiclass data sets. Through the analyses of experimental results, PBG is shown to outperform the other compared methods on all data sets in various aspects including the issues of data scarcity, dense-majority, DCDS, DCDD, and unscaled data.