Neural Architecture Search (NAS) is an open and challenging problem in machine learning. While NAS offers great promise, the prohibitive computational demand of most of the existing NAS methods makes it difficult to directly search the architectures on large-scale tasks. The typical way of conducting large scale NAS is to search for an architectural building block on a small dataset (either using a proxy set from the large dataset or a completely different small scale dataset) and then transfer the block to a larger dataset. Despite a number of recent results that show the promise of transfer from proxy datasets, a comprehensive evaluation of different NAS methods studying the impact of different source datasets has not yet been addressed. In this work, we propose to analyze the architecture transferability of different NAS methods by performing a series of experiments on large scale benchmarks such as ImageNet1K and ImageNet22K. We find that: (i) The size and domain of the proxy set does not seem to influence architecture performance on the target dataset. On average, transfer performance of architectures searched using completely different small datasets (e.g., CIFAR10) perform similarly to the architectures searched directly on proxy target datasets. However, design of proxy sets has considerable impact on rankings of different NAS methods. (ii) While different NAS methods show similar performance on a source dataset (e.g., CIFAR10), they significantly differ on the transfer performance to a large dataset (e.g., ImageNet1K). (iii) Even on large datasets, random sampling baseline is very competitive, but the choice of the appropriate combination of proxy set and search strategy can provide significant improvement over it. We believe that our extensive empirical analysis will prove useful for future design of NAS algorithms.

Large-scale distributed training of deep acoustic models plays an important role in today's high-performance automatic speech recognition (ASR). In this paper we investigate a variety of asynchronous decentralized distributed training strategies based on data parallel stochastic gradient descent (SGD) to show their superior performance over the commonly-used synchronous distributed training via allreduce, especially when dealing with large batch sizes. Specifically, we study three variants of asynchronous decentralized parallel SGD (ADPSGD), namely, fixed and randomized communication patterns on a ring as well as a delay-by-one scheme. We introduce a mathematical model of ADPSGD, give its theoretical convergence rate, and compare the empirical convergence behavior and straggler resilience properties of the three variants. Experiments are carried out on an IBM supercomputer for training deep long short-term memory (LSTM) acoustic models on the 2000-hour Switchboard dataset. Recognition and speedup performance of the proposed strategies are evaluated under various training configurations. We show that ADPSGD with fixed and randomized communication patterns cope well with slow learners. When learners are equally fast, ADPSGD with the delay-by-one strategy has the fastest convergence with large batches. In particular, using the delay-by-one strategy, we can train the acoustic model in less than 2 hours using 128 V100 GPUs with competitive word error rates.

Decentralized Parallel SGD (D-PSGD) and its asynchronous variant Asynchronous Parallel SGD (AD-PSGD) is a family of distributed learning algorithms that have been demonstrated to perform well for large-scale deep learning tasks. One drawback of (A)D-PSGD is that the spectral gap of the mixing matrix decreases when the number of learners in the system increases, which hampers convergence. In this paper, we investigate techniques to accelerate (A)D-PSGD based training by improving the spectral gap while minimizing the communication cost. We demonstrate the effectiveness of our proposed techniques by running experiments on the 2000-hour Switchboard speech recognition task and the ImageNet computer vision task. On an IBM P9 supercomputer, our system is able to train an LSTM acoustic model in 2.28 hours with 7.5% WER on the Hub5-2000 Switchboard (SWB) test set and 13.3% WER on the CallHome (CH) test set using 64 V100 GPUs and in 1.98 hours with 7.7% WER on SWB and 13.3% WER on CH using 128 V100 GPUs, the fastest training time reported to date.

Accurately and globally mapping human infrastructure is an important and challenging task with applications in routing, regulation compliance monitoring, and natural disaster response management etc.. In this paper we present progress in developing an algorithmic pipeline and distributed compute system that automates the process of map creation using high resolution aerial images. Unlike previous studies, most of which use datasets that are available only in a few cities across the world, we utilizes publicly available imagery and map data, both of which cover the contiguous United States (CONUS). We approach the technical challenge of inaccurate and incomplete training data adopting state-of-the-art convolutional neural network architectures such as the U-Net and the CycleGAN to incrementally generate maps with increasingly more accurate and more complete labels of man-made infrastructure such as roads and houses. Since scaling the mapping task to CONUS calls for parallelization, we then adopted an asynchronous distributed stochastic parallel gradient descent training scheme to distribute the computational workload onto a cluster of GPUs with nearly linear speed-up.

In this paper, we propose and investigate a variety of distributed deep learning strategies for automatic speech recognition (ASR) and evaluate them with a state-of-the-art Long short-term memory (LSTM) acoustic model on the 2000-hour Switchboard (SWB2000), which is one of the most widely used datasets for ASR performance benchmark. We first investigate what are the proper hyper-parameters (e.g., learning rate) to enable the training with sufficiently large batch size without impairing the model accuracy. We then implement various distributed strategies, including Synchronous (SYNC) , Asynchronous Decentralized Parallel SGD (ADPSGD) and the hybrid of the two HYBRID, to study their runtime/accuracy trade-off. We show that we can train the LSTM model using ADPSGD in 14 hours with 16 NVIDIA P100 GPUs to reach a 7.6% WER on the Hub5-2000 Switchboard (SWB) test set and a 13.1% WER on the Call-Home (CH) test set. Furthermore, we can train the model using HYBRID in 11.5 hours with 32 NVIDIA V100 GPUs without loss in accuracy.

Modern Automatic Speech Recognition (ASR) systems rely on distributed deep learning to for quick training completion.To enable efficient distributed training, it is imperative that the training algorithms can converge with a large mini-batch size.In this work, we discovered that Asynchronous Decentralized Parallel Stochastic Gradient Descent (ADPSGD) can work with much larger batch size than commonly used Synchronous SGD (SSGD) algorithm.On commonly used public SWB-300 and SWB-2000 ASR datasets, ADPSGD can converge with a batch size 3X as large as the one used in SSGD, thus enable training at a much larger scale.Further, we proposed a Hierarchical-ADPSGD (H-ADPSGD) system in which learners on the same computing node construct a super learner via a fast allreduce implementation, and super learners deploy ADPSGD algorithm among themselves.On a 64 Nvidia V100 GPU cluster connected via a 100Gb/s Ethernet network, our system is able to train SWB-2000 to reach a 7.6% WER on the Hub5-2000 Switchboard (SWB) test-set and a 13.2% WER on the Callhome (CH) test-set in 5.2 hours.To the best of our knowledge, this is the fastest ASR training system that attains this level of model accuracy for SWB-2000 task to be ever reported in the literature.

A fundamental shift in the technology has occurred at 90 nm CMOS and beyond where the interconnect resistance has been increasing so much that the distance a clock cycle can reach has been dwindling as a fraction of the dimension of the chip. to cause a repeater explosion problem. This problem translates into an explosion of repeaters which not only added significant overhead in area but also power, as repeaters are major contributors to leakage. By reaching out to the vertical dimension, 3D technology has the potential of easing repeater explosion (Figure 1), reducing latency between units, increasing memory bandwidth and integrating heterogeneous technologies. However, in order to exploit the full potential of 3D technology, new challenges in the area of system level design and analysis, physical design, thermal analysis need to be addressed.

Net weighting is a key technique in timing-driven placement (TDP), which plays a crucial role for deep submicron very large scale integration of physical synthesis and timing closure. A popular way to assign net weight is based on its slack, such that the worst negative slack (WNS) of the entire circuit may be minimized. While WNS is an important optimization metric, another figure of merit (FOM), defined as the total slack difference compared to a certain slack threshold for all timing end points, is of equal importance to measure the overall timing closure result for highly complex modern application specific integrated circuits and microprocessor designs. Moreover, to optimally assign net weight for timing closure, the effect of net weighting on timing should be carefully studied. In this paper, we perform a comprehensive analysis of the wirelength, slack, and FOM sensitivities to the net weight, and propose a new net weighting scheme based on those sensitivities. Such sensitivity analysis implicitly takes potential physical synthesis effect into consideration. The experiments on a set of industrial circuits show promising results for both stand-alone TDP and physical synthesis afterwards.

In this paper, we discuss timing closure for high performance microprocessor designs. Aggressive cycle time and deep sub-micron technology scaling introduce a myriad of problems that are not present in the ASIC domain. The impact of these problems on floorplanning, placement, clocking and logic synthesis is described. We present ideas and potential solutions for tackling these problems.

Net weighting is a key technique in large scale timing driven placement, which plays a crucial role for deep submicron physical synthesis and timing closure. A popular way to assign net weight is based on the slack of the nets, trying to minimize the worst negative slack (WNS) for the entire circuit. While WNS is an important optimization metric, another figure of merit (FOM), defined as the total slack difference compared to a certain slack threshold for all timing end points, is of equivalent importance to measure the overall timing closure result for highly complex modern ASIC and microprocessor designs. In this paper, we perform a comprehensive analysis of the slack and FOM sensitivities to the net weight, and propose a new net weighting scheme based on the slack and FOM sensitivities. Such sensitivity analysis implicitly takes potential physical synthesis effect into consideration. Experiment results on a set of industrial circuits are promising for both stand-alone timing driven placement and physical synthesis afterwards.

Power dissipation is becoming the most challenging design constraint in nanometer technologies. Among various design implementation schemes, standard cell ASICs offer the best power efficiency for high-performance applications. The flexibility of ASICs allow for the use of multiple voltages and multiple thresholds to match the performance of critical regions to their timing constraints, and minimize the power everywhere else. We explore the trade-off between multiple supply voltages and multiple threshold voltages in the optimization of dynamic and static power. The use of multiple supply voltages presents some unique physical and electrical challenges. Level shifters need to be introduced between the various voltage regions. Several level shifter implementations will be shown. The physical layout needs to be designed to ensure the efficient delivery of the correct voltage to various voltage regions. More flexibility can be gained by using appropriate level shifters. We will discuss optimization techniques such as clock skew scheduling which can be effectively used to push performance in a power neutral way.

Interconnect delays represent an increasingly dominant portion of overall circuit delays. During timing-driven physical synthesis process, timing analysis is repeatedly performed over several hundred thousand components. Thus, fast and accurate estimation of interconnect delays is crucial. Traditionally, lumped and elmore delay models have been widely used for computing interconnect delays in physical synthesis due to their computational efficiency. However, these delay models are known to be inaccurate since they ignore slew and resistive shielding effects. In this paper, we propose a new iterative refinement based delay estimation approach that considers resistive shielding along with driver slew. Experimental results show that the proposed approach gives not only highly accurate results for far end RC-line delays but also compares very favorably to more difficult to match source end delays and source slews. In addition, use of the proposed delay model in physical synthesis yields significant performance improvement on several large industrial ASICs.

The effectiveness of logic synthesis to satisfy increasingly tight timing constraints in deep-submicron high-performance circuits heavily depends on the range and variety of logic gates available in the standard cell library. Primarily, research in the design of high-performance standard cell libraries has been focused on drive strength selection of various logic gates. Since CMOS logic circuit delays not only depend on the drive strength of each gate but also on its PM width ratio, it is crucial to provide good PM width ratios for each cell. The main contribution of this paper is the development of a theoretical framework through which library designers can determine "optimal" PM width ratio for each logic gate in their high-performance standard cell library. This theoretical framework utilizes new gate delay models that explicitly represent the dependence of delay on P/N width ratio and load. These delay models yield highly accurate delay for CMOS gates in a 0.12 /spl mu/m L/sub eff/ deep-submicron technology.

This paper presents a gain-based fanout optimization algorithm for near-continuous buffer libraries. A near-continuous buffer library contains many buffers in a wide range of discrete sizes and each buffer of a specific type satisfies a size-independent delay equation. The new fanout algorithm is derived from an optimal algorithm to a special fanout optimization problem for continuous libraries. The gain-based technique constructs fanout trees which have better timing at similar area cost. Since no combinatorial search over buffer sizes or fanout tree topologies is used, our execution time is up to 1000 times faster when compared to conventional fanout algorithms.

This paper presents an algorithm to select a good set of gate sizes for the primitive gates of a standard cell library. A measurement error on a gate is dened to quantify the discrepancy resulting from replacing the size required by a synthesis sizing algorithm with a size available in a discrete cell library. The criterion for gate size selection is a set of gate sizes that minimizes the cumulative error of a prescribed measurement. Optimal solutions to the gate size selection problem targetting size and delay measurements are presented for cases when the probability distribution and the delay equations are simple. A realistic probability distribution is obtained using a sample space of gates derived fromagroup of designs that is synthesized under the semi-custom synthesis methodology [1]. A \\delay-match " (minimizing delay error) and a \\sizematch" (minimizing size error) set of gate sizes are obtained numerically, and are subsequently realized as discrete cell libraries. The previous group of designs are synthesized using the two selected cell libraries and two other cell libraries, one with \\equal-spacing " of cell sizes and the other with \\exponential-spacing " of cell sizes. The \\size-match " library gives the best overall slack and area results. 1

Hazard-non-increasing optimization algorithms, optimizations on gate-level logic without introduction of any further static nor dynamic hazards, are presented. Proofs are given for general theoretical results on hazard-non-increasing transformations which serve as the basis for these algorithms. The algorithms substantially augment the set of proven hazard-non-increasing optimization techniques in the literature. These algorithms are useful for hazard-free implementations of asynchronous designs.>