Examining thyroid fine needle aspiration (FNA) can grade cancer risks, derive prognostic information, and guide follow-up care or surgery decision-making. However, thyroid cytology's diagnostic cues are more dispersed compared with pathology images in other disciplines, making standard annotation strategy for AI diagnosis labor-intensive. Inspired by how cytologists diagnose under the microscope, we propose an innovative cellularity-based active learning framework, namely Cyto-AL, to correlate cellularity with diagnostic categories for the active learning query. We also improve the Whole Slide Image (WSI) category of The Bethesda System for Reporting Thyroid Cytology (TBSRTC) prediction by proposing severe-stage pinpointed Multiple Instance Learning (MIL). Additionally, we introduce a lightweight score model to optimize the query in human in the loop (HITL) annotation strategy. Given scarce public thyroid cytology datasets, we release our collected and labeled images as benchmarks. The benchmark comprises 138 WSIs (27,496 valid image patches) collected from 2021-2023 across six classes, annotated by three pathologists using TBSRTC. At patch-level verification, Cyto-AL achieves a 2.2% average classification accuracy improvement over state-of-the-art methods with an equally labeled dataset, and its lightweight ranking-aware model reduces training time by around 65%. Moreover, the WSI-level MIL approach improves average accuracy by 10.7% and Macro-F1 score by 3.5%, outperforming standard sampling methods such as Monte Carlo sampling. The source code and dataset are available at https://github.com/Junchao-Zhu/Cyto-AL.

Foundation Models (FMs) have shown great promise for multimodal medical image analysis such as Magnetic Resonance Imaging (MRI). However, certain MRI sequences may be unavailable due to various constraints, such as limited scanning time, patient discomfort, or scanner limitations. The absence of certain modalities can hinder the performance of FMs in clinical applications, making effective missing modality imputation crucial for ensuring their applicability. Previous approaches, including generative adversarial networks (GANs), have been employed to synthesize missing modalities in either a one-to-one or many-to-one manner. However, these methods have limitations, as they require training a new model for different missing scenarios and are prone to mode collapse, generating limited diversity in the synthesized images. To address these challenges, we propose DiffM4RI, a diffusion model for many-to-many missing modality imputation in MRI. DiffM4RI innovatively formulates the missing modality imputation as a modality-level inpainting task, enabling it to handle arbitrary missing modality situations without the need for training multiple networks. Experiments on the BraTs datasets demonstrate DiffM4RI can achieve an average SSIM improvement of 0.15 over MustGAN, 0.1 over SynDiff, and 0.02 over VQ-VAE-2. These results highlight the potential of DiffM4RI in enhancing the reliability of FMs in clinical applications. The code is available athttps://github.com/27yw/DiffM4RI.

Virtual staining can visualize transparent tissue and cellular structures in pathological images. However, existing virtual staining methods focus on emulating existing real-world stains. Furthermore, these methods are based on learning-based approaches, which can be computationally expensive and timeconsuming. To address these challenges, we propose a novel non-learning-based virtual staining framework that leverages physicsinspired particle swarm optimization to discover enhanced staining templates. Our method formulates virtual staining as an optimization problem, efficiently exploring the vast staining template space using a modified update strategy inspired by Newton's laws of motion. To guide the search process, we introduce an innovative composite staining quality metric that integrates colorfulness, tissue component differentiation, and perceptual quality. Our approach aims to balance visual vibrancy with perceptual fidelity. We evaluated our method on four diverse pathological image datasets, demonstrating consistent improvements in segmentation and classification performance compared to traditional staining methods, with performance gains ranging from 1.38 % to 2.03 %. The optimized virtual stains identified by our method expand beyond the limitations of existing real-world stains, offering new possibilities for tissue visualization and analysis.

Artificial intelligence (AI) has profoundly transformed our lives, reshaping industries and impacting nearly every aspect of society over the past few decades. It has recently become even more influential, primarily due to the rise of foundation models representing a new paradigm in AI development. These models, characterized by their large-scale training on vast datasets, have unique capabilities such as emergence and transference, enabling them to generalize across diverse tasks. Since their introduction, foundation models have been increasingly applied in fields such as autonomous driving, computer vision, marketing, finance, industrial robotics, and healthcare. Pathologists worldwide use computational methods to analyze diseases that profoundly impact human well-being, including cancer diagnosis and staging, genetic mutation prediction, and treatment and prognosis forecasting. In this article, we discuss how, despite the promise of foundation models in various applications, their development and application in computational pathology remain challenging due to inherent characteristics such as emergence, homogenization, hallucination, transference, compositionality, and explainability. While powerful, these traits introduce numerous ethical concerns and challenges, impacting safety and reliability, patient privacy, accountability, and equity and fairness in healthcare access. We examine these ethical issues, focusing on key concerns like algorithmic discrimination and misuse, accuracy, privacy breaches, transparency, public accessibility, and accountability. Furthermore, potential solutions to these challenges are analyzed, offering future perspectives on promoting the development and application of more ethical AI and foundation models in computational pathology. These insights aim to guide foundation models toward responsible integration of AI in healthcare.

Foundation models like the Segment Anything Model (SAM) have shown promising performance in general image segmentation tasks. However, their effectiveness is limited when applied to pathology images due to the inherent multi-scale structural complexity and staining heterogeneity. To address these challenges, we introduce SegAnyPath, a foundational model specifically designed for pathology image segmentation. SegAnyPath is trained on an extensive public pathology dataset comprising over 1.5 million images and 3.5 million masks. We propose a multi-scale proxy task to handle the diverse resolutions in pathology images, complementing the reconstruction objective in the supervised learning stage. To enhance segmentation performance across stain variations, we introduce a novel self-distillation scheme based on stain augmentations. Furthermore, we propose an innovative task-guided Mixture of Experts (MoE) architecture in the decoder of SegAnyPath for efficient management of distinct pathology segmentation tasks, including cell, tissue, and tumor segmentation. Experimental results demonstrate SegAnyPath's zero-shot generalization capability, achieving a Dice score of 0.6797 across multiple datasets and organs while maintaining consistent performance across varying staining styles and resolutions. In comparison, the fine-tuned SAM achieves a Dice score of only 0.5258 on the same external test sets, indicating a substantial 29.27% improvement by SegAnyPath. SegAnyPath has the potential to advance the field of pathology analysis and improve diagnostic accuracy in clinical settings. The code is available at https://github.com/wagnchogn/SegAnyPath.

The apparition of multimodal solutions helped closing this breach by teaching AI solutions how to use multiple inputs to infer a given task. Elevating AI algorithms to a higher level by increasing robustness and accuracy. This approach also made models more human, by learning from multiple sources just as how clinicians do. However, such models are still far from being flexible due to mostly be constrained into one singular task, vastly differing from real clinicians who are in charge of broad fields with multiple tasks to perform. This niche is perfect for AI algorithms to grow. However, existing AI models lack flexibility to act similarly to doctors. Thus, some innovations are required to overcome this rigidity. One possible solution would be the emergence of Medical Artificial Intelligence (GMAI) seeks to evolve the current paradigm by aspiring to emulate more humanlike intelligence in medical applications. The emergence of these "Virtual Doctors" has the potential to significantly transform the healthcare paradigm, offering convenient healthcare solutions with just a few clicks at any time of the day. The focal point of this paper is to delineate the prospective applications of GMAI within the field of telemedicine, positioned as ‘teledoctors’, and to outline the existing technological gap between these innovative models and their conventional counterparts. More specifically, we envision GMAI as a ‘Virtual Doctor,’ functioning as an adaptable telemedicine system capable of addressing a broad array of inquiries and providing diagnoses for a diverse range of medical conditions by utilizing patient- or doctor-provided data inputs. While a general AI model for medical imaging may offer some benefits, it is vital to consider such a model’s potential drawbacks and limitations. It is crucial to balance the benefits of AI and the need for human expertise and oversight in medical imaging.

Tabular data plays a crucial role in various domains but often suffers from missing values, thereby curtailing its potential utility. Traditional imputation techniques frequently yield suboptimal results and impose substantial computational burdens, leading to inaccuracies in subsequent modeling tasks. To address these challenges, we propose DiffImpute, a novel Denoising Diffusion Probabilistic Model (DDPM). Specifically, DiffImpute is trained on complete tabular datasets, ensuring that it can produce credible imputations for missing entries without undermining the authenticity of the existing data. Innovatively, it can be applied to various settings of Missing Completely At Random (MCAR) and Missing At Random (MAR). To effectively handle the tabular features in DDPM, we tailor four tabular denoising networks, spanning MLP, ResNet, Transformer, and U-Net. We also propose Harmonization to enhance coherence between observed and imputed data by infusing the data back and de-noising them multiple times during the sampling stage. To enable efficient inference while maintaining imputation performance, we propose a refined non-Markovian sampling process that works along with Harmonization. Empirical evaluations on seven diverse datasets underscore the prowess of DiffImpute. Specifically, when paired with the Transformer as the denoising network, it consistently outperforms its competitors, boasting an average ranking of 1.7 and the most minimal standard deviation. In contrast, the next best method lags with a ranking of 2.8 and a standard deviation of 0.9. The code is available at https://github.com/Dendiiiii/DiffImpute.

The wide prevalence of staining variations in digital pathology presents a significant obstacle, often undermining the effectiveness of diagnosis and analysis. The current strategies to counteract this issue primarily revolve around Stain Normalization (SN) and Stain Augmentation (SA). Nonetheless, these methodologies come with inherent limitations. They struggle to adapt to the vast array of staining styles, tend to presuppose linear associations between color spaces, and often lead to unrealistic color transformations. In response to these challenges, we introduce RandStainNA++, a novel method seamlessly integrating SN and SA. This method exploits the versatility of random SN and SA within randomly selected color spaces, effectively managing variations for the foreground and background independently. By refining the transformations of staining styles for the foreground and background within a realistic scope, this strategy promotes the generation of more practical staining transformations during the training phase. Further enhancing our approach, we propose a unique self-distillation method. This technique incorporates prior knowledge of stain variation, substantially augmenting the generalization capability of the network. The striking results yield that, compared to conventional classification models, our method boosts performance by a significant margin of 16-25%. Furthermore, when juxtaposed with baseline segmentation models, the Dice score registers an increase of 0.06.

Cancer is widely recognized as the primary cause of mortality worldwide, and pathology analysis plays a pivotal role in achieving accurate cancer diagnosis. The intricate representation of features in histopathological images encompasses abundant information crucial for disease diagnosis, regarding cell appearance, tumor microenvironment, and geometric characteristics. However, recent deep learning methods have not adequately exploited geometric features for pathological image classification due to the absence of effective descriptors that can capture both cell distribution and gathering patterns, which often serve as potent indicators. In this paper, inspired by clinical practice, a Hierarchical Graph Pyramid Transformer (HGPT) is proposed to guide pathological image classification by effectively exploiting a geometric representation of tissue distribution which was ignored by existing state-of-the-art methods. First, a graph representation is constructed according to morphological feature of input pathological image and learn geometric representation through the proposed multi-head graph aggregator. Then, the image and its graph representation are feed into the transformer encoder layer to model long-range dependency. Finally, a locality feature enhancement block is designed to enhance the 2D local representation of feature embedding, which is not well explored in the existing vision transformers. An extensive experimental study is conducted on Kather-5K, MHIST, NCT-CRC-HE, and GasHisSDB for binary or multi-category classification of multiple cancer types. Results demonstrated that our method is capable of consistently reaching superior classification outcomes for histopathological images, which provide an effective diagnostic tool for malignant tumors in clinical practice.

To address the challenge of diagnosing histology images accurately, we propose a novel Transformer-based model, HistDeblur, to detect and eliminate blur in digitized microscope images. HistDeblur adopts a trustworthy deblurring pyramid architecture with a shared-weight structure for artifact restoration, ensuring image quality control. To the best of our knowledge, HistDeblur is the first attempt at a Transformer based model to detect and eliminate the blur in histology images. Uniquely, it incorporates an auxiliary branch in the decoder for explainable blurriness estimation, diverging from traditional pyramid models. A method is also introduced to synthesize non-uniform blurry images, simulating real-world artifacts occurring during specimen preparation and digitization. Experiments on a representative subset of CRC-VAL-HE-7K [1] show the proposed framework is superior to the state-of-the-art approaches in the context of histology deblurring by a margin of 1.97 PSNR and 0.03 SSIM respectively. HistDeblur, being fully automated, optimizes histology image classification, quality assessment, and restoration, thereby mimicking the histologist’s examination experience proficiently. The approach facilitates reliable medical diagnoses, allowing histologists to make informed decisions on accepting deblurred images or opting for costly resampling and re-scanning. The source code is available at https://github.com/KKK-Liu/HistDeblur

This paper introduces a novel parallel relaxed flooding (PRF) algorithm for Voronoi diagram generation. The algorithm takes a set of reference points extracted from an image as input and assigns each GPU thread a partition of the image domain to perform parallel flooding computation. Our PRF algorithm has three advantages as follows. (1) The PRF algorithm divides an image domain into subregions for concurrent flooding computation. To achieve high parallelism, a point selection method is incorporated to remove dependencies between different subregions. (2) We exploit the sparsity of the input point data with a k-d tree. With the k-d tree data structure, the point selection step achieves high efficiency, and the amount of CPU-GPU data transfer is reduced. (3) We propose a relaxed flooding method, which achieves more accurate results and decreases memory traffic compared to the traditional flooding method. In addition to these advantages, we provide an empirical method to determine the appropriate parameter in the point selection step for high performance, given an expected error rate. We evaluated the performance of our method on multiple datasets. Compared with the state-of-the-art parallel banding algorithm, our method achieved an average speed-up of 4.6× on the randomly generated datasets with a point density of 0.01%, and 6.8× on nuclei segmentation datasets. The code of the PRF algorithm is publicly available*.

Currently, data-driven based machine learning is considered one of the best choices in clinical pathology analysis, and its success is subject to the sufficiency of digitized slides, particularly those with deep annotations. Although centralized training on a large data set may be more reliable and more generalized, the slides to the examination are more often than not collected from many distributed medical institutes. This brings its own challenges, and the most important is the assurance of privacy and security of incoming data samples. In the discipline of histopathology image, the universal stain-variation issue adds to the difficulty of an automatic system as different clinical institutions provide distinct stain styles. To address these two important challenges in AI-based histopathology diagnoses, this work proposes a novel conditional Generative Adversarial Network (GAN) with one orchestration generator and multiple distributed discriminators, to cope with multiple-client based stain-style normalization. Implemented within a Federated Learning (FL) paradigm, this framework well preserves data privacy and security. Additionally, the training consistency and stability of the distributed system are further enhanced by a novel temporal self-distillation regularization scheme. Empirically, on large cohorts of histopathology datasets as a benchmark, the proposed model matches the performance of conventional centralized learning very closely. It also outperforms state-of-the-art stain-style transfer methods on the downstream Federated Learning image classification task, with an accuracy increase of over 20.0% in comparison to the baseline classification model.

The artifacts in histology images may encumber the accurate interpretation of medical information and cause misdiagnosis. Accordingly, prepending manual quality control of artifacts considerably decreases the degree of automation. To close this gap, we propose a methodical pre-processing framework to detect and restore artifacts, which minimizes their impact on downstream AI diagnostic tasks. First, the artifact recognition network AR-Classifier first differentiates common artifacts from normal tissues, e.g., tissue folds, marking dye, tattoo pigment, spot, and out-of-focus, and also catalogs artifact patches by their restorability. Then, the succeeding artifact restoration network AR-CycleGAN performs de-artifact processing where stain styles and tissue structures can be maximally retained. We construct a benchmark for performance evaluation, curated from both clinically collected WSIs and public datasets of colorectal and breast cancer. The functional structures are compared with state-of-the-art methods, and also comprehensively evaluated by multiple metrics across multiple tasks, including artifact classification, artifact restoration, downstream diagnostic tasks of tumor classification and nuclei segmentation. The proposed system allows full automation of deep learning based histology image analysis without human intervention. Moreover, the structure-independent characteristic enables its processing with various artifact subtypes. The source code and data in this research are available at https://github.com/yunboer/AR-classifier-and-AR-CycleGAN.

Numerous patch-based methods have recently been proposed for histological image based breast cancer classification. However, their performance could be highly affected by ignoring spatial contextual information in the whole slide image (WSI). To address this issue, we propose a novel hierarchical Graph V-Net by integrating 1) patch-level pre-training and 2) context-based fine-tuning, with a hierarchical graph network. Specifically, a semi-supervised framework based on knowledge distillation is first developed to pre-train a patch encoder for extracting disease-relevant features. Then, a hierarchical Graph V-Net is designed to construct a hierarchical graph representation from neighboring/similar individual patches for coarse-to-fine classification, where each graph node (corresponding to one patch) is attached with extracted disease-relevant features and its target label during training is the average label of all pixels in the corresponding patch. To evaluate the performance of our proposed hierarchical Graph V-Net, we collect a large WSI dataset of 560 WSIs, with 30 labeled WSIs from the BACH dataset (through our further refinement), 30 labeled WSIs and 500 unlabeled WSIs from Yunnan Cancer Hospital. Those 500 unlabeled WSIs are employed for patch-level pre-training to improve feature representation, while 60 labeled WSIs are used to train and test our proposed hierarchical Graph V-Net. Both comparative assessment and ablation studies demonstrate the superiority of our proposed hierarchical Graph V-Net over state-of-the-art methods in classifying breast cancer from WSIs. The source code and our annotations for the BACH dataset have been released at https://github.com/lyhkevin/Graph-V-Net.

Histopathological identification of tumor tissue is one of the routine pathological diagnoses for pathologists. Recently, computational pathology has been successfully interpreted by a variety of deep learning-based applications. Nevertheless, the high-efficient and spatial-correlated processing of individual patches have always attracted attention in whole-slide image (WSI) analysis. In this paper, we propose a high-throughput system to detect tumor regions in colorectal cancer histology slides precisely. We train a deep convolutional neural network (CNN) model and design a Monte Carlo (MC) adaptive sampling method to estimate the most representative patches in a WSI. Two conditional random field (CRF) models are designed, namely the correction CRF and the prediction CRF are integrated for spatial dependencies of patches. We use three datasets of colorectal cancer from The Cancer Genome Atlas (TCGA) to evaluate the performance of the system. The overall diagnostic time can be reduced from 56.7 percent to 71.7 percent on the slides of a varying tumor distribution, with an increase in classification accuracy.

Pathology analysis is crucial to precise cancer diagnoses and the succeeding treatment plan as well. To detect abnormality in histopathology images with prevailing patch-based convolutional neural networks (CNNs), contextual information often serves as a powerful cue. However, as whole-slide images (WSIs) are characterized by intense morphological heterogeneity and extensive tissue scale, a straightforward visual span to a larger context may not well capture the information closely associated with the focal patch. In this paper, we propose a novel pixel-offset based patch-location method to identify high-representative tissues, with a CNN backbone. Pathology Deformable Conditional Random Field (PDCRF) is proposed to learn the offsets and weights of neighboring contexts in a spatial-adaptive manner, to search for high-representative patches. A CNN structure with the localized patches as training input is then capable of consistently reaching superior classification outcomes for histology images. Overall, the proposed method has achieved state-of-the-art performance, in terms of the test classification accuracy improvement to the baseline by 1.15-2.60%, 0.78-1.78%, and 1.47-2.18% on TCGA public datasets of TCGA-STAD, TCGA-COAD, and TCGA-READ respectively. It also achieves 88.95% test accuracy and 0.920 test AUC on Camelyon 16. To show the effectiveness of the proposed framework on downstream tasks, we take a further step by incorporating an active learning model, which noticeably reduces the number of manual annotations by PDCRF to reach a parallel patch-based histology classifier.

The massive annotation workload has always hindered the progress towards an automatic analysis of gigapixel whole-slide images. Histologically, individual patches from a constrained spatial region may share rich phenotypic information, where the morphological correlations have the potential to be mined for a grouping or clustering task. In this paper, we propose a clustering technique to extract multiple mutual information from histology images without prior domain knowledge. Specifically, our framework automatically localizes morphologically homogeneous patches within an extended solution space. Our novelty is an expanse and the pattern with which invariant information can be learnt, in contrast to the current literature of feature generation or parametric transformation within an individual patch. Additionally, structure-independent, the model may be applicable to any backbone convolutional neural network architectures. The empirical validation on The Cancer Genome Atlas (TCGA) datasets illustrates an observable margin of patch-level classification accuracy in comparison with state-of-the-art unsupervised approaches.

Expensive annotation cost has always been a critical obstacle in deep learning systems, in particular for the applications requiring domain experts’ knowledge, such as medical image analysis. Active learning has attracted widespread attention by decreasing the quantity and difficulty in annotation with query strategies. To make a query for labeling, contextual features of candidates’ relation are often considered essential. In this paper, we propose an innovative method that incorporating spatial distribution approximation in the uncertainty sampling for whole-slide histopathology image annotation. The active query selection combines the measure of spatial representativeness and histopathological tissue informativeness. With the assumption that the labeling cost of individual instances is non-biased, we use three independent features, namely loss prediction query, Monte Carlo distribution query, and loss estimation enhanced by spatial information for the active learning task. The experiments were validated on large cohorts of The Cancer Genome Atlas (TCGA) and the ”100,000 histological images of human colorectal cancer and healthy tissue” dataset. Empirically, the proposed method can outperform the other annotation strategies on the histopathology datasets. The annotation cost is reduced by an obvious margin of 40% to retain an accurate classifier. This novel annotation strategy provides the potential to efficiently label and classify histopathology images with a patch-based convolutional neural network.

Microsatellite instability (MSI) is the result of a defective DNA mismatch repair (MMR) system, and its presence occurs in a variety of cancers. The determination of MSI in colorectal cancer (CRC) will have a better prognosis and management of cancer patients. As the routine MSI identification via molecular testing is expensive, time-consuming, and region-restricted, novel methods to detect MSI are of great interest. In this work, we propose a multi-stage convolutional neural network (CNN) based framework to identify MSI status in colorectal cancer patients from histopathological images. A mislabel-aware module is designed to deal with the uncertainty problem in global-local labelling. An auto-grading model is proposed to discriminate patches by the degree of their histopathological correlation with recognizable MSI status, and subsequently aggregate the weights to make slide-level predictions. Our proposed methodology outperforms the existing models in the classification accuracy, and explicitly sorts out patches with representative features. The research outcome has the potential to assist in the interpretation of histopathology as a surrogate for MSI testing and also in the study of recognizable morphology of MSI-H/MSS tumors. Furthermore, this approach can be extended and applied to other cancer types.

To tailor for DNN (Deep Neural Network) acceleration, GPU has migrated to new architectures such as NVIDIA Volta and Turing that incorporate dedicated Tensor Cores. Although good at GEMM (generic matrix-matrix multiplication), Tensor Cores still have inefficiency facing convolutions with certain layer structures. This paper proposes a GPNPU (General-Purpose Neural-network Processing Unit) architecture, which offers another option of direct convolution in GPU. It stitches the direct convolution dataflow into the Tensor Cores with little hardware support, and resorts to regulated data layout with stripe-mined convolution execution to achieve higher performance and power efficiency, while retaining the general programability as GPU. We further apply a unified core design to support varied operand types and precision for higher computing throughput. The evaluation shows that GPNPU can outperform Tensor Cores on typical DNNs by 1.4X for inference (FP16) and 1.2X for training with much reduced power. The INT8 performance even increases to 2.4X. Our study demonstrates that it is possible and appealing to refine the Tensor Cores for greater DNN acceleration, while conforming to GPU architecture for the programmability necessary in future DNN evolution.

In the last decade, computational pathology has attracted notable attention in the deep learning domain. However, even on the state-of-the-art deep learning computing platforms, a high-resolution scanned whole slide image (WSI) still requires reducing into massive patches to be processed, which is very time consuming in real-time diagnosis. In this paper, we propose a high-throughput tumor location system with Monte Carlo adaptive sampling to accelerate WSI analysis. Additionally, we design a dynamic programming framework to incorporate spatial correlation, which can iteratively eliminate false positives or false negatives in the identification or tumor tissues. We use three datasets of colorectal cancer from The Cancer Genome Atlas (TCGA) for performance evaluation. The designed computer-aided system can reduce more than 50% of the diagnostic time on average in the tumor location task, along with a slight increase in accuracy.

Nowadays, high-definition video object recognition (segmentation and detection) is not within the easy reach of a real-time task in a consumer SoC due to the limited on-chip computing power for neural network (NN) processing. Although many accelerators have been optimized heavily, they are still isolated from the intrinsic video compression expertise in a decoder. Given the fact that a great portion of frames can be dynamically reconstructed by a few key frames with high fidelity in a video, we envision that the recognition can also be reconstructed in a similar way so as to save a large amount of NN computing power. In this paper, we study the feasibility and efficiency of a novel decoder-assisted NN accelerator architecture for video recognition (VR-DANN) in a conventional SoC-styled design, which for the first time tightly couples the working principle of a video decoder with the NN accelerator to provide smooth high-definition video recognition experience. We leverage motion vectors, the simple tempo-spatial information already available in the decoding process to facilitate the recognition process, and propose a lightweight NN-based refinement scheme to suppress the non-pixel recognition noise. We also propose the corresponding microarchitecture design, which can be built upon any existing commercial IPs with minimal hardware overhead but significant speedup. Our experimental results show that the VR-DANN-parallel architecture achieves 2.9× performance improvement with less than 1% accuracy loss compared with the state-of-the-art "FAVOS" scheme widely used for video recognition. Compared with optical flow assisted "DFF" scheme, it can achieve 2.2× performance gain and 3% accuracy improvement. As to another "Euphrates" scheme, VR-DANN can achieve 40% performance gain and comparable accuracy.

Training Convolutional Neural Networks(CNNs) is both memory-and computation-intensive. The resistive random access memory (ReRAM) has shown its advantage to accelerate such tasks with high energy-efficiency. However, the ReRAM-based pipeline architecture suffers from the low utilization of computing resource, caused by the imbalanced data throughput in different pipeline stages because of the inherent down-sampling effect in CNNs and the inflexible usage of ReRAM cells. In this paper, we propose a novel ReRAM-based bidirectional pipeline architecture, named HUBPA, to accelerate the training with higher utilization of the computing resource. Two stages of the CNN training, forward and backward propagations, are scheduled in HUBPA dynamically to share the computing resource. We design an accessory control scheme for the context switch of these two tasks. We also propose an efficient algorithm to allocate computing resource for each neural network layer. Our experiment results show that, compared with state-of-the-art ReRAM pipeline architecture, HUBPA improves the performance by 1.7X and reduces the energy consumption by 1.5X, based on the current benchmarks.

With GPUs heading towards general-purpose, hardware caching, e.g. the first-level data (L1D) cache is introduced into the on-chip memory hierarchy for GPGPUs. However, facing the GPGPU massive multi-threading, the small L1D requires a better management for a higher hit rate to benefit the performance. In this paper, on observing the L1D usage inefficiency, such as data duplication among streaming multiprocessors (SMs) that wastes the precious L1D resources, we first propose a shared L1.5D cache that substitutes the private L1D caches in several SMs to reduce the duplicated data and in turn increase the effective cache size for each SM. We evaluate and adopt a suitable layout of L1.5D to meet the timing requirements in GPGPUs. Then, to protect the sharable data from early evictions, we propose a sharable data aware cache management, which leverages a lightweight PC-based history table to protect sharable data on cache replacement. The experiments demonstrate that the proposed design can achieve an averaged 20.1% performance improvement with an increased on-chip hit rate by 16.9% for applications with sharable data.

It is a trend now that computing power through parallelism is provided by multi-core systems or heterogeneous architectures for High Performance Computing (HPC) and scientific computing. Although many algorithms have been proposed and implemented using sequential computing, alternative parallel solutions provide more suitable and high performance solutions to the same problems. In this paper, three parallelization strategies are proposed and implemented for a dynamic programming based cloud smoothing application, using both shared memory and non-shared memory approaches. The experiments are performed on NVIDIA GeForce GT750m and Tesla K20m, two GPU accelerators of Kepler architecture. Detailed performance analysis is presented on partition granularity at block and thread levels, memory access efficiency and computational complexity. The evaluations described show high approximation of results with high efficiency in the parallel implementations, and these strategies can be adopted in similar data analysis and processing applications.