Due to the limited resources of end devices, the task of Convolutional Neural Network (CNN) inference on the end-side is moving towards edge-end collaboration. However, existing collaborative methods mainly focus on offloading CNN inference tasks from end devices to a single-edge server, which leads to inefficient use of computational resources among nearby edge servers. Moreover, offloading the CNN inference task to a single third-party server may raise privacy concerns. To address these challenges, we propose a framework named AdapCP that introduces a collaborative and adaptive parallel acceleration strategy that utilizes the end device and multiple edge servers. AdapCP consists of two stages: (1) offloading to nearby servers and (2) parallel processing of the CNN inference. For the offloading phase, we use integer linear programming to find the partition points at the inter-layer level. For the parallel phase, we first investigate intra-layer structural splitting methods tailored for both convolutional and fully connected layers. Then, we employ a Deep Deterministic Policy Gradient (DDPG) algorithm based on the Dirichlet distribution to decide the partition points. Finally, we set a periodic update index to enhance AdapCP’s adaptability to dynamic environments. Empirical evaluations conducted on the Jetson nano demonstrate that AdapCP significantly reduces the total latency of CNN inference by an average factor of 2.21 \(\times\) compared to existing solutions.

The Sparse General Matrix-Matrix multiplication (SpGEMM) is a fundamental component for many applications, such as algebraic multigrid methods (AMG), graphic processing, and deep learning. However, the unbearable latency of computing high-dimensional, large-scale sparse matrix multiplication on GPUs hinders the development of these applications. An effective approach is heterogeneous cores collaborative computing, but this method must address three aspects: (1) irregular non-zero elements lead to load imbalance and irregular memory access, (2) different core computing latency differences reduce computational parallelism, and (3) temporary data transfer between different cores introduces additional latency overhead. In this work, we propose an innovative framework for collaborative large-scale sparse matrix multiplication on CPU-GPU heterogeneous cores, named ApSpGEMM. ApSpGEMM is based on sparsity rules and proposes reordering and splitting algorithms to eliminate the impact of non-zero element distribution features on load and memory access. Then adaptive panels allocation with affinity constraints among cores improves computational parallelism. Finally, carefully arranged asynchronous data transmission and computation balance communication overhead. Compared with state-of-the-art SpGEMM methods, our approach provides excellent absolute performance on matrices with different sparse structures. On heterogeneous cores, the GFlops of large-scale sparse matrix multiplication is improved by 2.25 to 7.21 times.

The Vision Transformer (ViT), which benefits from utilizing self-attention mechanisms, has demonstrated superior accuracy compared to CNNs. However, due to the expensive computational costs, deploying and inferring ViTs on resource-constrained mobile devices has become a challenge. To resolve this challenge, we conducted an empirical analysis to identify performance bottlenecks in deploying ViTs on mobile devices and explored viable solutions. In this paper, we propose SPViT, an adaptive split and offloading method that accelerates ViT inference on mobile devices. SPViT executes collaborative inference of ViT across available edge devices. We introduce a fine-grained splitting technique for the vision transformer structure. Furthermore, we propose an algorithm based on the Auto Regression model to predict partition latency and adaptive offload partitions. Finally, we design offline and online optimization methods to minimize the computational and communication overhead on each device. Based on real-world prototype experiments, SPViT effectively reduces inference latency by 2.2x to 3.3x across four state-of-the-art models.