While pre-trained language models like BERT have achieved impressive results on various natural language processing tasks, deploying them on resource-restricted devices is challenging due to their intensive computational cost and memory footprint. Previous approaches mainly focused on training smaller versions of a BERT model with competitive accuracy under limited computational resources. In this paper, we extend Length Adaptive Transformer and propose to design Token and Head Adaptive Transformer, which can compress and accelerate various BERT-based models via simple fine-tuning. We train a transformer with a progressive token and head pruning scheme, eliminating a large number of redundant tokens and attention heads in the later layers. Then, we conduct a multi-objective evolutionary search with the overall number of floating point operations (FLOPs) as its efficiency constraint to find joint token and head pruning strategies that maximize accuracy and efficiency under various computational budgets. Empirical studies show that a large portion of tokens and attention heads could be pruned while achieving superior performance compared to the baseline BERT-based models and Length Adaptive Transformers in various downstream NLP tasks. MobileBERT trained with our joint token and head pruning scheme achieves a GLUE score of 83.0, which is 1.4 higher than Length Adaptive Transformer and 2.9 higher than the original model.

Accurately measuring the absolute depth of every pixel captured by an imaging sensor is of critical importance in real-time applications such as autonomous navigation, augmented reality and robotics. In order to predict dense depth, a general approach is to fuse sensor inputs from different modalities such as LiDAR, camera and other time-of-flight sensors. LiDAR and other time-of-flight sensors provide accurate depth data but are quite sparse, both spatially and temporally. To augment missing depth information, generally RGB guidance is leveraged due to its high resolution information. Due to the reliance on multiple sensor modalities, design for robustness and adaptation is essential. In this work, we propose a transformer-like self-attention based generative adversarial network to estimate dense depth using RGB and sparse depth data. We introduce a novel training recipe for making the model robust so that it works even when one of the input modalities is not available. The multi-head self-attention mechanism can dynamically attend to most salient parts of the RGB image or corresponding sparse depth data producing the most competitive results. Our proposed network also requires less memory for training and inference compared to other existing heavily residual connection based convolutional neural networks, making it more suitable for resource-constrained edge applications. The source code is available at: https://github.com/kocchop/robust-multimodal-fusion-gan

Predicting dense depth accurately is essential for 3D scene understanding applications such as autonomous driving and robotics. However, the depth obtained from commercially available LiDAR and Time-of-Flight sensors is very sparse. With RGB color guidance, modern convolutional neural network (CNN) based approaches can recover the missing depth information. However, there could be scenarios such as low-light environments where it might be difficult to get an associated RGB image with the sparse depth. In this work, we propose a Generative Adversarial Network (GAN) that can accurately predict the dense depth using only sparse samples without any RGB inputs. Generally, the sparsity in the depth samples is uniformly distributed and cannot guarantee capturing all intricate details. In this study, we also explore different variants of sparse sampling strategies from uniform to feature based directed sampling. We find that feature based intelligent sampling enjoys better compression ratio without sacrificing intricate details, saving data communication bandwidth. Compared to uniform sampling, depending on how aggressively the directed sampling is done, we observe about 3% to 25% reduction in size. We can easily reduce the size by 8% with directed sampling without sacrificing the reconstruction accuracy. Although such directed sampling strategies are not readily available with commercially viable depth sensors, we believe that our study paves the way for future intelligent sensing and sampling strategies. To further investigate data reduction and reconstruction accuracy trade-offs we deploy our GAN to generate higher resolution dense depth from 4 times smaller sparse samples. With slight decrease in accuracy, our GAN is able to recover the depth successfully which shows great promise in edge Internet of Things (IoT) applications where we have very tight constraint on data transmission bandwidth. Our source code along with examples is available at: https://github.com/kocchop/depth-completion-gan

Reducing the model size and computation costs for dedicated AI accelerator designs, neural network quantization methods have attracted momentous attention recently. Unfortunately, merely minimizing quantization loss using constant discretization causes accuracy deterioration. In this paper, we propose an iterative accuracy-driven learning framework of competitive-collaborative quantization (CCQ) to gradually adapt the bit-precision of each individual layer. Orthogonal to prior quantization policies working with full precision for the first and last layers of the network, CCQ offers layer-wise competition for any target quantization policy with holistic layer fine-tuning to recover accuracy, where the state-of-the-art networks can be entirely quantized without any significant accuracy degradation.