Convolutional neural networks (CNNs) have long been the paradigm of choice for robust medical image processing (MIP). Therefore, it is crucial to effectively and efficiently deploy CNNs on devices with different computing capabilities to support computer-aided diagnosis. Many methods employ factorized convolutional layers to alleviate the burden of limited computational resources at the expense of expressiveness. To this end, given weak medical image-driven CNN model optimization, a Singular value equalization generalizer-induced Factorized Convolution (SFConv) is proposed to improve the expressive power of factorized convolutions in MIP models. We first decompose the weight matrix of convolutional filters into two low-rank matrices to achieve model reduction. Then minimize the KL divergence between the two low-rank weight matrices and the uniform distribution, thereby reducing the number of singular value directions with significant variance. Extensive experiments on fundus and OCTA datasets demonstrate that our SFConv yields competitive expressiveness over vanilla convolutions while reducing complexity.

The characteristics of 3D nodules on Computed Tomography (CT), including size, location, shape, and attenuation, are primary medical clues for distinguishing between benign and malignant nodules. To support evidence-based decision-making for lung cancer screening in clinical practice, we present a 3D Nodule Content-based Metric Learning (3D-NCML) network to retrieve subsolid-benign, subsolid-malignant, solid-benign, and solid-malignant nodules similar to the indeterminate ones. The inputs of 3D-NCML are 3D patches that exactly contain the whole nodule to ensure all visual information is included. A spatial position and size coding module, a shape encoder module, and an attenuation extraction module are designed based on medical clues for guiding the network to learn important characteristics of nodules. Experiments on the LIDC-IDRI dataset and a private dataset demonstrate that 3D-NCML outperforms other methods by quantitative and qualitative analysis, with more similar nodules retrieved and ranked ahead.

With modeling on triangle similarity, margin-based triplet (MT) loss functions have been the paradigm of choice for robust deep metric learning (DML). However, they require carefully tuning a violation margin being the task-specific decision boundary. A globally invariant violation margin is irrational when performing online triplet mining on each mini-batch. To address this issue, we propose a novel yet efficient concordance-induced triplet (CIT) loss function for DML training to specify an individualized violation margin for each triplet. Concordance expects the predicted ordering of intra-class and inter-class similarities to be correct. Building on the concordance of triangle similarity, our CIT loss maximizes an intra-class similarity relative to two inter-class similarities. In addition, due to the high training complexity on triplets, we propose a regularizer for our CIT loss, called spectral decay (SD), to enhance data-driven DML model training in a parameter-driven manner. SD exploits the rank-one approximation of the spectral norm direction in the weight matrix to impose varying weight decay, thus collaborating with CIT loss to affect weight variance shifts to achieve fast convergence. Extensive experiments on various DML tasks, including face recognition, person re-identification, and image retrieval, demonstrate the elegance and superiority of our CIT against its counterparts.

The growth rate of pulmonary nodules is a critical clue to the cancerous diagnosis. It is essential to monitor their dynamic progressions during pulmonary nodule management. To facilitate the prosperity of research on nodule growth prediction, we organized and published a temporal dataset called NLSTt with consecutive computed tomography (CT) scans. Based on the self-built dataset, we develop a visual learner to predict the growth for the following CT scan qualitatively and further propose a model to predict the growth rate of pulmonary nodules quantitatively, so that better diagnosis can be achieved with the help of our predicted results. To this end, in this work, we propose a parameterized Gempertz-guided morphological autoencoder (GM-AE) to generate any future-time-span high-quality visual appearances of pulmonary nodules from the baseline CT scan. Specifically, we parameterize a popular mathematical model for tumor growth kinetics, Gompertz, to predict future masses and volumes of pulmonary nodules. Then, we exploit the expected growth rate on the mass and volume to guide decoders generating future shape and texture of pulmonary nodules. We introduce two branches in an autoencoder to encourage shape-aware and textural-aware representation learning and integrate the generated shape into the textural-aware branch to simulate the future morphology of pulmonary nodules. We conduct extensive experiments on the self-built NLSTt dataset to demonstrate the superiority of our GM-AE to its competitive counterparts. Experiment results also reveal the learnable Gompertz function enjoys promising descriptive power in accounting for inter-subject variability of the growth rate for pulmonary nodules. Besides, we evaluate our GM-AE model on an in-house dataset to validate its generalizability and practicality. We make its code publicly available along with the published NLSTt dataset.

Existing multi-view learning methods based on the information bottleneck principle exhibit impressing generalization by capturing inter-view consistency and complementarity. They leverage cross-view joint information (consistency) and view-specific information (complementarity) while discarding redundant information. By fusing visual features, multi-view learning methods help medical image processing to produce more reliable predictions. However, multi-views of medical images often have low consistency and high complementarity due to modal differences in imaging or different projection depths, thus challenging existing methods to balance them to the maximal extent. To mitigate such an issue, we improve the information bottleneck (IB) loss function with a balanced regularization term, termed IBB loss, reassembling the constraints of multi-view consistency and complementarity. In particular, the balanced regularization term with a unique trade-off factor in IBB loss helps minimize the mutual information on consistency and complementarity to strike a balance. In addition, we devise a triplet multi-view network named TM net to learn the consistent and complementary features from multi-view medical images. By evaluating two datasets, we demonstrate the superiority of our method against several counterparts. The extensive experiments also confirm that our IBB loss significantly improves multi-view learning in medical images.

It turns out that convolutional neural networks (CNNs) have excellent medical image processing capabilities. Hence, effectively and efficiently deploying CNNs on devices with varying computing power to make computer-aided diagnosis puts on the agenda. However, it is a dilemma to balance the limited computing resources and model complexity. Previously, we proposed factorized convolution with spectral normalization (FConvSN) to mitigate the bottleneck of deploying CNNs for 2D medical images. But due to the cube structure of 3D convolutional kernels, it does not work well for 3D medical images. Directly flattening 3D kernels to 2D weights for matrix factorization may undermine the learning ability along depth-wise, resulting in the loss of depth information and the decline of model performance. To this end, we factorize a 3D convolutional kernel to 2D weight matrices with depth-wise dimensions, then assign an attentive score for each 2D weight matrix by a depth-wise dependencies-induced adaptive attention block (AA). AA with a temperature hyper-parameter helps convolution kernel to better capture depth-wise dependencies in 3D medical images, improving its learning ability along the depth direction. We term this novel factorized convolution as FConvAA used for compressing model complexity without impairing the depth-wise expressivity. We also impose spectral normalization (SN) for FConvAA to constrain spectral norm-wise weights. We conduct extensive experiments on the public lung CT dataset LUNA16 and the private retina OCT dataset to demonstrate the effectiveness and feasibility of our FConvAA.

When encountering a dubious diagnostic case, medical instance retrieval can help radiologists make evidence-based diagnoses by finding images containing instances similar to a query case from a large image database. The similarity between the query case and retrieved similar cases is determined by visual features extracted from pathologically abnormal regions. However, the manifestation of these regions often lacks specificity, i.e., different diseases can have the same manifestation, and different manifestations may occur at different stages of the same disease. To combat the manifestation ambiguity in medical instance retrieval, we propose a novel deep framework called Y-Net, encoding images into compact hash-codes generated from convolutional features by feature aggregation. Y-Net can learn highly discriminative convolutional features by unifying the pixel-wise segmentation loss and classification loss. The segmentation loss allows exploring subtle spatial differences for good spatial-discriminability while the classification loss utilizes class-aware semantic information for good semantic-separability. As a result, Y-Net can enhance the visual features in pathologically abnormal regions and suppress the disturbing of the background during model training, which could effectively embed discriminative features into the hash-codes in the retrieval stage. Extensive experiments on two medical image datasets demonstrate that Y-Net can alleviate the ambiguity of pathologically abnormal regions and its retrieval performance outperforms the state-of-the-art method by an average of 9.27% on the returned list of 10.

Deep hashing methods have been proved to be effective for the large-scale medical image search assisting reference-based diagnosis for clinicians. However, when the salient region plays a maximal discriminative role in ophthalmic image, existing deep hashing methods do not fully exploit the learning ability of the deep network to capture the features of salient regions pointedly. The different grades or classes of ophthalmic images may be share similar overall performance but have subtle differences that can be differentiated by mining salient regions. To address this issue, we propose a novel end-to-end network, named Attention-based Saliency Hashing (ASH), for learning compact hash-code to represent ophthalmic images. ASH embeds a spatial-attention module to focus more on the representation of salient regions and highlights their essential role in differentiating ophthalmic images. Benefiting from the spatial-attention module, the information of salient regions can be mapped into the hash-code for similarity calculation. Extensive experiments on two different modalities of ophthalmic image datasets demonstrate that the proposed ASH can further improve the retrieval performance compared to the state-of-the-art deep hashing methods due to the huge contributions of the spatial-attention module.

Latent Factor Model (LFM) is one of the most successful methods for Collaborative filtering (CF) in the recommendation system, in which both users and items are projected into a joint latent factor space. Base on matrix factorization applied usually in pattern recognition, LFM models user-item interactions as inner products of factor vectors of user and item in that space and can be efficiently solved by least square methods with optimal estimation. However, such optimal estimation methods are prone to overfitting due to the extreme sparsity of user-item interactions. In this paper, we propose a Bayesian treatment for LFM, named Bayesian Latent Factor Model (BLFM). Based on observed user-item interactions, we build a probabilistic factor model in which the regularization is introduced via placing prior constraint on latent factors, and the likelihood function is established over observations and parameters. Then we draw samples of latent factors from the posterior distribution with Variational Inference (VI) to predict expected value. We further make an extension to BLFM, called BLFMBias, incorporating user-dependent and item-dependent biases into the model for enhancing performance. Extensive experiments on the movie rating dataset show the effectiveness of our proposed models by compared with several strong baselines.

Nuclear cataract is one of the most common types of cataract. In the recent, ophthalmologists are increasingly using anterior segment optical coherence tomography (AS-OCT) images to diagnose many ocular diseases including cataract. The relationship between cataract and the lens opacity based on AS-OCT images has been being studied in clinical pioneer research. However, using AS-OCT images to classify cataract automatically based on computer-aided diagnosis (CAD) technique has not been seriously studied. This paper proposes a novel Convolutional Neural Network (CNN) model named GraNet for nuclear cataract classification based on AS-OCT images. In the GraNet, we introduce a grading block to learn high-level feature representations based on the pointwise convolution method. To further improve the classification performance, we propose a simple and efficient cross-training method is comprised of focal loss and cross-entropy loss. Extensive experiments are conducted on the AS-OCT image dataset, the results demonstrate that the proposed methods achieve better nuclear cataract classification results than baselines.

As a natural extension of image-based cross-modal recipe retrieval, retrieving a specific video given a recipe as the query is seldom explored. There are various temporal and spatial elements hidden in cooking videos. In addition, current image-based cross-modal recipe retrieval approaches mostly emphasize the understanding of textual and visual content independently. Such methods overlook the interaction between textual and visual content. In this work, we innovatively propose a new problem of video-based cross-modal recipe retrieval and thoroughly investigate this issue under the attention paradigm. In particular, we firstly exploit a parallel-attention network to independently learn the representations of videos and recipes. Next, a co-attention network is proposed to explicitly emphasize the cross-modal interactive features between videos and recipes. Meanwhile, a cross-modal fusion sub-network is proposed to learn both the independent and collaborative dynamics, which can enhance the associated representation of videos and recipes. Last but not the least, the embedding vectors of videos and recipes stemming from joint network are optimized with a pairwise ranking loss. Extensive experiments on a self-collected dataset have verified the effectiveness and rationality of our proposed solution.

In online education systems, the courseware plays a pivotal role in helping educators present and impart knowledge to students. The originality of courseware heavily impacts the choice of educators, because the teaching content evolves and so does courseware. However, how to measure the originality of a courseware is a challenging task, due to the lack of labels and the difficulty of quantification. To this end, we contribute a similarity ranking-based unsupervised approach to measure the originality of a courseware. In particular, we first exploit a pre-trained deep visual-text embedding to obtain the representations of images and texts in a local manner. Next, inspired by the design of capsule neural network, a vector-based pooling network is proposed to learn multimodal representations of images and texts. Finally, we propose a Discriminator to optimize the model by maximizing the mutual information between local features and global features in an unsupervised manner. To evaluate the performance of our proposed model, we further subtly collect a dataset for evaluating the originality of courseware by treating sequential versions of each courseware as ranking lists. Therefore, the learning-to-rank scheme can be utilized to evaluate the similarity-based ranking performance. Extensive experimental results have demonstrated the superiority of our proposed framework as compared to other state-of-the-art competitors.