The Google Edge TPU is an ASIC designed to accelerate inference of deep learning models on edge devices. Edge TPU only supports a limited set of operations. In those deep learning models containing unsupported operations, the Edge TPU compiler maps the unsupported operations and their succeeding operations to execute on the CPU, even if the succeeding operations can be executed on the Edge TPU. As a result, the Edge TPU is under-utilized and the performance significantly degrades. To overcome this issue, we have developed a model rewriting tool, which leverages MLIR to replace unsupported operations in the model with supported ones while maintaining the same functionality. We also propose a general method to approximate arbitrary continuous functions to any precision using the ReLU operation. Experimental results show that our transformation achieves an average speedup of 1.66× and 4.44× over the models without rewriting on the server and edge platforms, respectively.

Video captioning is a core technology to many important applications, such as AI-assisted medical diagnosis, video question answering, storytelling through videos, and lip-reading. Video captioning employs a hybrid CNN + RNN model. Accelerating such a hybrid model on a heterogeneous system is challenging for two reasons. First, CNN and RNN exhibit very different computing behaviors, making the mapping between computation and heterogeneous devices difficult. Second, data dependency exists between the CNN and RNN within a video frame and between adjacent RNNs across video frames. These data dependencies prohibit the full parallelization of the hybrid model. The issues also include the utilization of accelerator resources, which is critical to maximizing the performance. In this work, we propose a fine-grained scheduling scheme for mapping computation and devices within a video frame, and a pipeline scheduling scheme for exploiting maximum parallelism between the execution of the video frames. In addition, we propose two capacity-guided scheduling methods. On the server, the concurrent kernel execution mechanism is exploited for improving GPU utilization. On the edge platform, we rearrange CNN computation among the CPU and EdgeTPUs guided by the EdgeTPU’s SRAM capacity so that balanced computation is achieved and off-chip memory overhead is minimized. Experimental results show that our scheduling scheme improves video captioning performance by up to 3.24 \( \times \) with CPU + GPU collaboration over the GPU-only execution. On an edge platform with an ARM CPU and two EdgeTPUs, our CPU + EdgeTPU scheduling exhibits outstanding performance, which achieves up to 54.9 \( \times \) speedup compared to using ARM CPU only and can perform video captioning of 59 frames per second.