Quantization is a neural network compression technique that effectively improves the deployment performance on inference hardware. Fixed-point quantization methods use the same bit-width for all layers in the network, which leads to difficulties in balancing compression rate and accuracy loss. Therefore, mixed-precision quantization has recently been proposed. The major challenge of the mixed-precision quantization is to select the quantization bit-widths of each layer in network to simultaneously meet the requirements of minimizing accuracy loss and hardware resource consumption. In this paper, we present a Single-Step Hardware-Aware Quantization (SHQ) method. It can calculate the resource consumption of hardware such as Field Programmable Gate Arrays (FPGAs) before actual deployment and find effective quantization schemes by just single step, different from common software-hardware two-step approaches with large workload and time-consuming. Genetic algorithm is combined with SHQ to search quantization schemes with low hardware resource usage and high accuracy. In addition, the correlation between hardware resource cost and qualitative indicators of proxy signal has been analyzed to bring computer-aided design insights of neural network accelerators. Experiments of full-pipeline accelerator deployment on FPGAs platforms show that our approach saves 39% of Digital Signal Processors (DSP) and 59% of Block Random Access Memory (BRAM) usage on MobileNet compared to full 8-bit quantization, while the accuracy drops by only 0.56%. The source code about our method can be found at this link:https://github.com/hujie369/SHQ.

Edge computing for artificial intelligence (AI) has become a new focus today. At the edge, the growing complexity and diversity of AI models has made FPGA, which has shorter development cycles, a good choice. Traditional AI accelerators on FPGA are mainly based on the Compute Engine (CE) architecture, suffering from low resource utilization and suboptimal speed. In contrast, the pipeline architecture achieves higher performance through its algorithm-structure-aware feature and fully on-chip data flow. However, customized designs and large bandwidth demands bring new challenges to its development agility and memory utilization, while high-performance acceleration for complex neural networks is still hard. In this article, we proposed MetaAccel, a novel fully pipelined accelerator. It uses two clock domains to manage data scheduling and calculation, significantly improving computing resource efficiency and on-chip memory utilization. Besides that, we built a hyperparameter-driven resource estimation model that can match the most appropriate design solutions for specific network structures. Based on this architecture, processing method for networks with complex branch structures and various operations is given, which makes MetaAccel suitable for Convolutional Neural Networks (CNNs) in different fields, such as image classification, object detection, and image segmentation. For typical networks, MetaAccel can achieve a throughput of more than 0.7TOPS and a DSP efficiency of up to 2.0GOPS/DSP and outperforms previous FPGA work in other metrics, showing its advantages in complex CNNs’ acceleration.

The rapid development of artificial intelligence (AI) has met people’s personalized needs. However, with the increase of data capacities and computing requirements, the imbalance between large-scale data transmission and limited network bandwidth has become increasingly prominent. To improve the speed of embedded system, real-time intelligent computing is gradually moving from the cloud to the edge. Traditional FPGA-based AI accelerators mainly utilize PE architecture, but the low computing throughput and resource utilization make it difficult to meet the power requirement of edge AI application scenarios such as image segmentation. In recent years, AI accelerators based on streaming architecture have become a trend, and it is necessary to customize high-performance streaming accelerators for specific segmentation algorithms. In this paper, we design a high-intensity pixel-level fully pipelined accelerator with customized strategies to eliminate the sparse and redundant computations in specific algorithms of semantic segmentation, which significantly improve the accelerator’s computing throughput and hardware resources utilization. On Xilinx FPGA, our acceleration of two typical semantic segmentation networks-ESPNet and DeepLabV3, achieves optimized throughputs of 171.3 GOPS and 1324.8 GOPS, and computing efficiency of 9.26 and 9.01, respectively. It provides the possibility of hardware deployment in real-time application with high computing intensity.

In the era of artificial intelligence (AI), rapidly growing data and computing demands stimulate a shift toward more intelligent processing at the edge in the Internet of Things (IoT). AI application scenarios, such as image segmentation, pose new challenges to the computing capability of edge hardware, which cannot be solved by traditional AI accelerators with the traditional processing element (PE) architecture. Recently, the streaming architecture has received more attention due to its higher performance. To improve the throughput of edge platforms for segmentation tasks, customizing streaming accelerators for segmentation models is now necessary. Based on these motivations, we proposed pipelined encoder-decoder segmentation model accelerator (PEDSA), a fully pipelined streaming accelerator for convolutional encoder-decoder segmentation networks. PEDSA maps all the layers in the network into a pixel-level pipeline. All operations, especially upsampling (unpooling, deconvolution, etc.) which involves complex data rearrangement, are integrated into a regular, concise, and fully on-chip data flow. On Xilinx field-programmable gate arrays (FPGAs), our acceleration of SegNet-Basic and U-Net reached performances of 2676.47 and 7646.29 GOPS, respectively, outperforming previous accelerators for this kind of network. This work provides new ideas for the deployment of segmentation algorithms at the edge.

The design of convolutional neural network (CNN) hardware accelerators based on a single computing engine (CE) architecture or multi-CE architecture has received widespread attention in recent years. Although this kind of hardware accelerator has advantages in hardware platform deployment flexibility and development cycle, it is still limited in resource utilization and data throughput. When processing large feature maps, the speed can usually only reach 10 frames/s, which does not meet the requirements of application scenarios, such as autonomous driving and radar detection. To solve the above problems, this article proposes a full pipeline hardware accelerator design based on pixel. By pixel-by-pixel strategy, the concept of the layer is downplayed, and the generation method of each pixel of the output feature map (Ofmap) can be optimized. To pipeline the entire computing system, we expand each layer of the neural network into hardware, eliminating the buffers between layers and maximizing the effect of complete connectivity across the entire network. This approach has yielded excellent performance. Besides that, as the pixel data stream is a fundamental paradigm in image processing, our fully pipelined hardware accelerator is universal for various CNNs (MobileNetV1, MobileNetV2 and FashionNet) in computer vision. As an example, the accelerator for MobileNetV1 achieves a speed of 4205.50 frames/s and a throughput of 4787.15 GOP/s at 211 MHz, with an output latency of 0.60 ms per image. This extremely shorts processing time and opens the door for AI's application in high-speed scenarios.

Language-based multi-modal learning has showcased remarkable efficacy across various domains. However, certain areas such as electrocardiogram (ECG) analysis face challenges due to incomplete and scarce textual data. Analyzing incomplete corpora poses difficulties for pre-trained language models, while data scarcity makes it difficult tre alignment and classification respectively, classification loss combined with consistent contexts are added in pre-training to alleviate gaps between these targets. Meanwhile, consistent contexts narrow gaps between prompts, allowing models to focus on genuinely informative features. Consequently, ECG feature spaces align more closely with semantic spaces, ensuring robust classification performance and enhancing the quality of multi-modal representations. Through extensive experiments involving zero-shot and few-shot learning, CELL demonstrates superior performances in near-distribution, out-of-distribution, and clinical domains. Its robust and generalized performance positions CELL as a promiso harmonize multimodal pretraining and downstream goals. To address these problems, this study introduces a novel ECG-Language multi-modal learning method named Context ECG-Language Learning (CELL). To supplement context and complete corpora, dynamic contexts composed of learnable vectors are incorporated into language model embedding, with other parameters fixed. Since pre-training and downstream targets are featuing approach for multi-modal learning in fields with scarce and incomplete corpora.

Recently, contrastive learning (CL) has garnered wide interest because it enables unsupervised pretraining to alleviate conventional deep learning methods’ strong reliance on artificial labels. While CL-based methods have been applied to cardiovascular disease (CVD) diagnosis with noninvasive electrocardiogram (ECG), most of these methods are limited within the 1-D signal modality and primarily focus on temporal features like amplitude and time sequence. The morphological features derived from clinically significant image-like ECGs are ignored. Furthermore, the relationships among different leads are neglected as well, describing the activities and interaction of various heart regions that are essential in CVD diagnosis and lesion localization. To address these limitations, this work proposes a novel cross-modal CL framework named signal–image–graph cross-modal contrastive learning (SIGxCL), which represents and jointly analyzes ECGs in signal, image, and graph modalities. Crucial for CL, modality-specific transformations are introduced for ECGs in the three modalities. SIGxCL enables signal–image–graph correspondence by maximizing the agreement of the accordant cross-modal ECGs in the invariant space. Consequently, SIGxCL could capture and leverage temporal, morphological, and spatially physiological features simultaneously. Compared to random initial and conventional supervised methods, SIGxCL achieves remarkable enhancements. Considering the best performances of the existing CL-based methods, SIGxCL outperforms them by up to 4.72%, 9.41%, and 4.31% across three data sets. SIGxCL is designed to be compatible with Internet of Medical Things (IoMT) and can be deployed on resource-limited portable devices. The deployment includes pretraining, online/offline tuning, and real-time inference modules. In conclusion, SIGxCL demonstrates superior performance and provides a promising approach for real-time IoMT-based diagnosis.

Recently, deep learning (DL) has enabled rapid advancements in electrocardiogram (ECG)-based automatic cardiovascular disease (CVD) diagnosis. Multi-lead ECG signals have lead systems based on the potential differences between electrodes placed on the limbs and the chest. When applying DL models, ECG signals are usually treated as synchronized signals arranged in Euclidean space, which is the abstraction and generalization of real space. However, conventional DL models typically merely focus on temporal features when analyzing Euclidean data. These approaches ignore the spatial relationships of different leads, which are physiologically significant and useful for CVD diagnosis because different leads represent activities of specific heart regions. These relationships derived from spatial distributions of electrodes can be conveniently created in non-Euclidean data, making multi-lead ECGs better conform to their nature. Considering graph convolutional network (GCN) adept at analyzing non-Euclidean data, a novel spatial-temporal residual GCN for CVD diagnosis is proposed in this work. ECG signals are firstly divided into single-channel patches and transferred into nodes, which will be connected by spatial-temporal connections. The proposed model employs residual GCN blocks and feed-forward networks to alleviate over-smoothing and over-fitting. Moreover, residual connections and patch dividing enable the capture of global and detailed spatial-temporal features. Experimental results reveal that the proposed model achieves at least a 5.85% and 6.80% increase inF1over other state-of-the-art algorithms with similar parameters and computations in both PTB-XL and Chapman databases. It indicates that the proposed model provides a promising avenue for intelligent diagnosis with limited computing resources.

The development of healthcare industry, especially Internet of Medical Things (IoMT), has generated considerable unlabeled electrocardiogram (ECG) signals. This article proposes a new unsupervised feature learning method for these unlabeled 12-lead ECGs, a type of 12-channel 1-D time series. Based on contrastive predictive coding (CPC), it considers the characteristics of 12-lead ECGs and develops novel lead-separation CPC (LSCPC) and lead-combination CPC (LCCPC). Specifically, LSCPC captures intralead features for each lead, while LCCPC combines all the leads and explores interlead relationships. Furthermore, a fusion model of LSCPC and LCCPC generates final representations. The Physikalisch-Technische Bundesanstalt (PTB)-XL database that contains 21837 12-lead records is used for unsupervised feature learning. Using learned features, linear classifiers are trained to accomplish the downstream tasks. 448 ECG records from 148 myocardial infarction (MI) and 52 healthy control subjects of the PTB database are used for MI detection. 6877 records from the CPSC-2018 database are used for atrial fibrillation (AF) detection, including 918 normal records, 1098 AF records, and 4861 other records. Using fivefold cross-validation, our model achieves 90.38% and 73.27% accuracy in MI and AF detection, respectively. Compared with existing models, it improves the performances by at least 2.32% for MI detection and 3.99% for AF detection. The model has been deployed on a lightweight embedded system (800-MHz ARM processor, 1-GB RAM). The maximum latency is only 465.34 ms, which can satisfy the real-time constraints. Overall, all the results have demonstrated the potential of our method for real-world healthcare, and lightweight IoMT applications.

This paper proposes a novel hybrid network named multiple-feature-branch convolutional bidirectional recurrent neural network (MFB-CBRNN) for myocardial infarction (MI) detection using 12-lead ECGs. The model efficiently combines convolutional neural network-based and recurrent neural network-based structures. Each feature branch consists of several one-dimensional convolutional and pooling layers, corresponding to a certain lead. All the feature branches are independent from each other, which are utilized to learn the diverse features from different leads. Moreover, a bidirectional long short term memory network is employed to summarize all the feature branches. Its good ability of feature aggregation has been proved by the experiments. Furthermore, the paper develops a novel optimization method, lead random mask (LRM), to alleviate overfitting and implement an implicit ensemble like dropout. The model with LRM can achieve a more accurate MI detection. Class-based and subject-based fivefold cross validations are both carried out using Physikalisch-Technische Bundesanstalt diagnostic database. Totally, there are 148 MI and 52 healthy control subjects involved in the experiments. The MFB-CBRNN achieves an overall accuracy of 99.90% in class-based experiments, and an overall accuracy of 93.08% in subject-based experiments. Compared with other related studies, our algorithm achieves a comparable or even better result on MI detection. Therefore, the MFB-CBRNN has a good generalization capacity and is suitable for MI detection using 12-lead ECGs. It has a potential to assist the real-world MI diagnostics and reduce the burden of cardiologists.

Neural networks are powerful computation tools for mimicking the human brain to solve realistic problems. Since spiking neural networks are a type of brain-inspired network, called the novel spiking system, Monitor-based Spiking Recurrent network (MbSRN), is derived to learn and represent patterns in this paper. This network provides a computational framework for memorizing the targets using a simple dynamic model that maintains biological plasticity. Based on a recurrent reservoir, the MbSRN presents a mechanism called a 'monitor' to track the components of the state space in the training stage online and to self-sustain the complex dynamics in the testing stage. The network firing spikes are optimized to represent the target dynamics according to the accumulation of the membrane potentials of the units. Stability analysis of the monitor conducted by limiting the coefficient penalty in the loss function verifies that our network has good anti-interference performance under neuron loss and noise. The results of solving some realistic tasks show that the MbSRN not only achieves a high goodness-of-fit of the target patterns but also maintains good spiking efficiency and storage capacity.

Error functions are normally based on the distance between output spikes and target spikes in supervised learning algorithms for spiking neural networks (SNNs). Due to the discontinuous nature of the internal state of spiking neuron, it is challenging to ensure that the number of output spikes and target spikes kept identical in multispike learning. This problem is conventionally dealt with by using the smaller of the number of desired spikes and that of actual output spikes in learning. However, if this approach is used, information is lost as some spikes are neglected. In this paper, a probability-modulated timing mechanism is built on the stochastic neurons, where the discontinuous spike patterns are converted to the likelihood of generating the desired output spike trains. By applying this mechanism to a probability-modulated spiking classifier, a probability-modulated SNN (PMSNN) is constructed. In its multilayer and multispike learning structure, more inputs are incorporated and mapped to the target spike trains. A clustering rule connection mechanism is also applied to a reservoir to improve the efficiency of information transmission among synapses, which can map the highly correlated inputs to the adjacent neurons. Results of comparisons between the proposed method and popular the SNN algorithms showed that the PMSNN yields higher efficiency and requires fewer parameters.

In this paper, a novel algorithm based on a convolutional neural network (CNN) is proposed for myocardial infarction detection via multilead electrocardiogram (ECG). A beat segmentation algorithm utilizing multilead ECG is designed to obtain multilead beats, and fuzzy information granulation is adopted for preprocessing. Then, the beats are input into our multilead-CNN (ML-CNN), a novel model that includes sub two-dimensional (2-D) convolutional layers and lead asymmetric pooling (LAP) layers. As different leads represent various angles of the same heart, LAP can capture multiscale features of different leads, exploiting the individual characteristics of each lead. In addition, sub 2-D convolution can utilize the holistic characters of all the leads. It uses 1-D kernels shared among the different leads to generate local optimal features. These strategies make the ML-CNN suitable for multilead ECG processing. To evaluate our algorithm, actual ECG datasets from the PTB diagnostic database are used. The sensitivity of our algorithm is 95.40%, the specificity is 97.37%, and the accuracy is 96.00% in the experiments. Targeting lightweight mobile healthcare applications, real-time analyses are performed on both MATLAB and ARM Cortex-A9 platforms. The average processing times for each heartbeat are approximately 17.10 and 26.75 ms, respectively, which indicate that this method has good potential for mobile healthcare applications.

To provide a fast compression algorithm for high‐resolution medical image sequences, an efficient three‐dimensional (3D) separate descendant‐based (SBD) set partitioning in hierarchical trees (SPIHT) algorithm (3D SDB‐SPIHT) is proposed in this study. To accelerate the transformation, 3D integer wavelet transform is used first. Based on an efficient spatial–temporal tree structure, which is designed for the transformed coefficients, the authors propose a fast coding scheme by separating the descendant set into offspring set and leaves set. The proposed algorithm has more selectivity in deciding the scanning and coding of the descendant sets and hence the coding time is accelerated. Experimental results demonstrate that 3D SDB‐SPIHT compresses medical images faster compared with traditional 3D SPIHT and other variations of 3D SPIHT.