Inverted bottleneck layers, which are built upon depth-wise convolutions, have been the predominant building blocks in state-of-the-art object detection models on mobile devices. In this work, we investigate the optimality of this design pattern over a broad range of mobile accelerators by revisiting the usefulness of regular convolutions. We discover that regular convolutions are a potent component to boost the latency-accuracy trade-off for object detection on accelerators, provided that they are placed strategically in the network via neural architecture search. By incorporating regular convolutions in the search space and directly optimizing the network architectures for object detection, we obtain a family of object detection models, MobileDets, that achieve state-of-the-art results across mobile accelerators. On the COCO object detection task, MobileDets outperform MobileNetV3+SSDLite by 1.7 mAP at comparable mobile CPU inference latencies. MobileDets also outperform MobileNetV2+SSDLite by 1.9 mAP on mobile CPUs, 3.7 mAP on Google EdgeTPU, 3.4 mAP on Qualcomm Hexagon DSP and 2.7 mAP on Nvidia Jetson GPU without increasing latency. Moreover, MobileDets are comparable with the state-of-the-art MnasFPN on mobile CPUs even without using the feature pyramid, and achieve better mAP scores on both EdgeTPUs and DSPs with up to 2× speedup. Code and models are available in the TensorFlow Object Detection API [16]: https://github.com/tensorflow/models/tree/master/research/object_detection.

Deep learning with a large number of parame-ters requires distributed training, where model accuracy and runtime are two important factors to be considered. However, there has been no systematic study of the tradeoff between these two factors during the model training process. This paper presents Rudra, a parameter server based distributed computing framework tuned for training large-scale deep neural networks. Using variants of the asynchronous stochastic gradient descent algorithm we study the impact of synchronization protocol, stale gradient updates, minibatch size, learning rates, and number of learners on runtime performance and model accuracy. We introduce a new learningrate modulation strategy to counter the effect of stale gradients and propose a new synchronization protocol that can effectively bound the staleness in gradients, improve runtime performance and achieve good model accuracy. Our empirical investigation reveals a principled approach for distributed training of neural networks: the mini-batch size per learner should be reduced as more learners are added to the system to preserve the model accuracy. We validate this approach using commonly-used image classification benchmarks: CIFAR10 and ImageNet.

editorial Free Access Share on Guest Editors’ Introduction: Hardware and Algorithms for On-Chip Learning Authors: Yu Cao Arizona State University, Tempe, Arizona Arizona State University, Tempe, ArizonaView Profile , Xin Li Carnegie Mellon University, Pittsburgh, PA Carnegie Mellon University, Pittsburgh, PAView Profile , Taemin Kim Intel, Hillsboro, OR Intel, Hillsboro, ORView Profile , Suyog Gupta Google, Mountain View, CA Google, Mountain View, CAView Profile Authors Info & Claims ACM Journal on Emerging Technologies in Computing SystemsVolume 13Issue 3July 2017 Article No.: 30pp 1–3https://doi.org/10.1145/3022193Published:09 February 2017Publication History 0citation335DownloadsMetricsTotal Citations0Total Downloads335Last 12 Months17Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

Deep learning with a large number of parametersrequires distributed training, where model accuracy and runtimeare two important factors to be considered. However, there hasbeen no systematic study of the tradeoff between these two factorsduring the model training process. This paper presents Rudra, aparameter server based distributed computing framework tunedfor training large-scale deep neural networks. Using variants ofthe asynchronous stochastic gradient descent algorithm we studythe impact of synchronization protocol, stale gradient updates, minibatch size, learning rates, and number of learners on runtimeperformance and model accuracy. We introduce a new learningrate modulation strategy to counter the effect of stale gradientsand propose a new synchronization protocol that can effectivelybound the staleness in gradients, improve runtime performanceand achieve good model accuracy. Our empirical investigationreveals a principled approach for distributed training of neuralnetworks: the mini-batch size per learner should be reducedas more learners are added to the system to preserve the modelaccuracy. We validate this approach using commonly-used imageclassification benchmarks: CIFAR10 and ImageNet.

Training of large-scale deep neural networks is often constrained by the available computational resources. We study the effect of limited precision data representation and computation on neural network training. Within the context of low-precision fixed-point computations, we observe the rounding scheme to play a crucial role in determining the network’s behavior during training. Our results show that deep networks can be trained using only 16-bit wide fixed-point number representation when using stochastic rounding, and incur little to no degradation in the classification accuracy. We also demonstrate an energy-efficient hardware accelerator that implements low-precision fixed-point arithmetic with stochastic rounding