Vector Quantization (VQ) is an appealing model compression method to obtain a tiny model with less accuracy loss. While methods to obtain better codebooks and codes under fixed clustering dimensionality have been extensively studied, optimizations via the reduction of subvector dimensionality are not carefully considered. This paper reports our recent progress on model compression with the combination of dimensionality reduction and vector quantization, proposing Low-Rank Representation Vector Quantization (LR2VQ). LR2VQ joins low-rank representation with subvector clustering to construct a new kind of building block that is optimized by end-to-end training. In our method, the compression ratio could be directly controlled by the dimensionality of subvectors, and the final accuracy is solely determined by clustering dimensionality$\tilde{d}$. We recognize$\tilde{d}$as a trade-off between low-rank approximation error and clustering error and carry out both theoretical analysis and experimental observations that empower the estimation of the proper$\tilde{d}$before fine-tuning. With a proper$\tilde{d}$, we evaluate LR2VQ with ResNet-18/ResNet-50 on ImageNet classification datasets, achieving 2.8%/1.0% top-1 accuracy improvements over the current state-of-the-art model compression algorithms with 43×/31× compression factor.

The choice of activation functions is crucial to deep neural networks. ReLU is a popular hand-designed activation function. Swish, the automatically searched activation function, outperforms ReLU on many challenging datasets. However, the search method has two main drawbacks. First, the tree-based search space is highly discrete and restricted, which is difficult to search. Second, the sample-based search method is inefficient in finding specialized activation functions for each dataset or neural architecture. To overcome these drawbacks, we propose a new activation function called Piecewise Linear Unit (PWLU), incorporating a carefully designed formulation and learning method. PWLU can learn specialized activation functions for different models, layers, or channels. Besides, we propose a non-uniform version of PWLU, which maintains sufficient flexibility but requires fewer intervals and parameters. Additionally, we generalize PWLU to three-dimensional space to define a piecewise linear surface named 2D-PWLU, which can be treated as a non-linear binary operator. Experimental results show that PWLU achieves SOTA performance on various tasks and models, and 2D-PWLU is better than element-wise addition when aggregating features from different branches. The proposed PWLU and its variation are easy to implement and efficient for inference, which can be widely applied in real-world applications.

Recently, PWLU has been proposed to learn specialized activation functions with straightforward piecewise linear definition and SOTA performance in different vision tasks and neural networks. However, the uniformly distributed intervals strongly limit the flexibility of PWLU, and the definition of PWLU requires the statistic-based realignment method to handle the misalignment between PWLU and input data. This paper proposes a new piecewise linear activation function called Nonuniform Piecewise Linear Unit (N-PWLU). N-PWLU has two advantages to overcome the drawbacks in PWLU. First, non-uniformly distributed intervals are used to increase flexibility. Second, the cumulative definition establishes close connections to the parameters in different intervals, which helps alleviate the misalignment issue. With these advantages, N-PWLU significantly outperforms PWLU, especially with fewer intervals. For example, on ImageNet classification dataset, 4-interval N-PWLU outperforms 4-interval PWLU with 1.15% top-1 accuracy in MobileNet-V3. Besides, the expressivity of 4-interval N-PWLU is compatible with 16-interval PWLU in different datasets and architectures. Fewer intervals simplify the computation of N-PWLU, which is friendly to be deployed on edge devices. We believe that our N-PWLU gets a step further in learning better parametric activation functions.

The choice of activation functions is crucial for modern deep neural networks. Popular hand-designed activation functions like Rectified Linear Unit(ReLU) and its variants show promising performance in various tasks and models. Swish, the automatically discovered activation function, has been proposed and outperforms ReLU on many challenging datasets. However, it has two main drawbacks. First, the tree-based search space is highly discrete and restricted, which is difficult for searching. Second, the sample-based searching method is inefficient, making it infeasible to find specialized activation functions for each dataset or neural architecture. To tackle these drawbacks, we propose a new activation function called Piecewise Linear Unit(PWLU), which incorporates a carefully designed formulation and learning method. It can learn specialized activation functions and achieves SOTA performance on large-scale datasets like ImageNet and COCO. For example, on ImageNet classification dataset, PWLU improves 0.9%/0.53%/1.0%/1.7%/1.0% top-1 accuracy over Swish for ResNet-18/ResNet-50/MobileNet-V2/MobileNetV3/EfficientNet-B0. PWLU is also easy to implement and efficient at inference, which can be widely applied in real-world applications.