Transformer-based architectures have recently emerged as the backbone for neural networks handling 3D point cloud applications, achieving state-of-the-art accuracy by avoiding the computational overhead of traditional k-nearest neighbor (k-NN) algorithms widely used in previous models. Despite their success, these transformer-based models often rely on submanifold convolution and attention layers, which introduce performance bottlenecks due to substantial weight parameters and high computational demands. In this paper, we propose two key hardware-friendly optimizations to address these challenges. First, we introduce a panel-level kernel mapping and weight-fetching strategy for weight-stationary computation, which minimizes data movement during submanifold convolution. Second, we propose a locality-based compression technique for attention queries and keys, allowing feature reuse from neighboring points to reduce the number of operations and memory requirements. Based on the optimizations, we develop a dedicated hardware tailored for 3D transformer-based 3D point clouds. Experimental results show that our approach significantly reduces memory accesses and enables efficient attention score computation with negligible accuracy loss.

Recently, point-based deep neural networks (DNN) have demonstrated remarkable ability in analyzing point cloud data. However, challenges arise in sampling and grouping layers, particularly in terms of time and energy consumption due to the iterative access and computation of point cloud data for local feature extraction. In this paper, we introduce a Morton code-based data structure which stores point data with the shared upper bits together, enabling sequential access to the points within a specific voxel. We also propose a fused sampling and grouping approach with a reduced search space, which reuses the point data and the calculated distances for the farthest voxel and its neighbors. Additionally, a dedicated hardware architecture is introduced to maximize the efficiency of the proposed optimization technique. Experimental results show that our approach effectively reduces the number of distance calculations and data accesses with negligible accuracy loss, without requiring retraining of the network model.

With the focus on three-dimensional (3D) applications, the importance of applying deep learning to point clouds have been growing recently. It is known that mapping operations including sampling and grouping play a critical role in extracting local features in point-based deep learning models. However, the mapping operations often become bottlenecks in terms of computing times due to the repetitive comparison of distances between input points. In this paper, we analyzed the characteristics of distance distribution during sampling and grouping operations, and discovered that substantial portion of the distance comparison does not need exact 3D Euclidean distance using all three coordinates. Based on the observations, we propose a technique called single coordinate comparison which selectively determines the comparison output with 1D-distance only. We also present a hardware architecture with a distance calculator capable of handling both 3D and 1D distance. The experimental results demonstrate the effectiveness of our approach in reducing both time and energy consumption, particularly as the number of points increases.