Abstract In the past few years, Transformer-based large language models (LLM) have become the dominant technology in a series of applications. To scale up the sequence length of the Transformer, FlashAttention is proposed to compute exact attention with reduced memory requirements and faster execution. However, implementing the FlashAttention algorithm on the new generation Sunway Supercomputer faces many constraints such as the unique heterogeneous architecture and the limited memory bandwidth. This work proposes SWattention, a highly efficient method for computing the exact attention on the SW26010pro processor. To fully utilize the 6 core groups (CG) and 64 cores per CG on the processor, we design a two-level parallel task partition strategy. Asynchronous memory access is employed to ensure that memory access overlaps with computation. Additionally, a tiling strategy is introduced to determine optimal SRAM block sizes. Compared with the standard attention, SWattention achieves around 2.0x speedup for FP32 training and 2.5x speedup for mixed-precision training. The sequence lengths range from 1k to 8k and scale up to 16k without being out of memory. As for the end-to-end performance, SWattention achieves up to 1.26x speedup for training GPT-style models, which demonstrates that SWattention enables longer sequence length for LLM training.

Aphids, brown spots, mosaics, rusts, powdery mildew and Alternaria blotches are common types of early apple leaf pests and diseases that severely affect the yield and quality of apples. Recently, deep learning has been regarded as the best classification model for apple leaf pests and diseases. However, these models with large parameters have difficulty providing an accurate and fast diagnosis of apple leaf pests and diseases on mobile terminals. This paper proposes a novel and real-time early apple leaf disease recognition model. AD Convolution is firstly utilized to replace standard convolution to make smaller number of parameters and calculations. Meanwhile, a LAD-Inception is built to enhance the ability of extracting multiscale features of different sizes of disease spots. Finally, the LAD-Net model is built by the LR-CBAM and the LAD-Inception modules, replacing a full connection with global average pooling to further reduce parameters. The results show that the LAD-Net, with a size of only 1.25MB, can achieve a recognition performance of 98.58%. Additionally, it is only delayed by 15.2ms on HUAWEI P40 and by 100.1ms on Jetson Nano, illustrating that the LAD-Net can accurately recognize early apple leaf pests and diseases on mobile devices in real-time, providing portable technical support.

Early detection of apple leaf diseases is the basis for timely precautions, which can inhibit the spread of the diseases and minimize the severe economic loss. Nowadays, CNN-based models are used for apple leaf diseases detection. However, due to the large model size and inference delay, the model is challenging to be transplanted to mobile terminals with good detection performance. This paper proposes a lightweight detection model Apple-YOLO on mobile terminals for real-time apple leaf diseases detection. First, a dataset named AppleSet8 is constructed using digital image processing and Mosaic data augmentation to improve the robustness and generalization ability of the model. Then the double-branch Apple-CSP module is presented to reduce the model parameters and guarantee feature extraction capability. Fur-thermore, the improved FDSA (Focus layer with depthwise separable convolution and attention mechanism) module effectively decreases the model's FLOPs and enhances the network's attention to the disease spots. Finally, the Skip-Spp (Skip-connection and Spatial pyramid pooling) module is built to strengthen the detection performance for multi-scale disease spots. The experiment results show that mobile-based Apple-YOLO has achieved 96.04% mAP, the inference speed of 34 FPS, and the size is only 5.33 ME, indicating that Apple-YOLO is suitable for the real-time detection of early apple leaf diseases in the real scenario.

At present, the identification of apple leaf diseases plays an important role in controlling apple leaf diseases and improving apple yield. CNNs(Convolutional Neural Networks) have been widely used in apple leaf diseases identification, but the training of the CNNs requires a large number of images. The lack of images would make the CNNs hard to generalize. Thus the CNNs are unable to recognize new disease images. Focusing on this problem, this paper proposes a new model named CGAN-IRB(Conditional Generative Adversarial Network with the Improved Residual Block) for data augmentation. Firstly, various improvements have been made based on CGAN to generate high-quality, robust, and specific-category images of apple leaf diseases. Among which the embedding of the residual block has been found to significantly improve the model performance. Then the interpolation algorithm is used instead of deconvolution to increase the image size. Finally, the TTUR(Two-Timescale Update Rule) training strategy is employed and all the convolutional layers of the network are spectrally normalized to stabilize the training of the network. The performance of CGAN-IRB was tested both on image generation and classification tasks. Experiment results show that the images generated by the network possess high quality and robust features, pro-viding a novel solution for the data augmentation of apple leaf diseases. The new GAN-based data augmentation method leads to significant improvements in the classification accuracy of CNNs. In the case of all tested CNNs, the classification accuracy improvements are 11.75% and 2.17% on average over non-augmented and traditional-augmented, respectively. Among them, the classification accuracy of GoogLeNet V2 and ShuffleNet V2 is 99.34% and 99.67%, respectively. The data augmentation approach proposed in this paper can be used more widely in the field of disease identification, solving the problem of insufficient data sets, and can be extended to related fields where data sets are difficult to obtain.