HW-aware neural architecture search (HW-NAS) aims to yield high-accuracy neural network (NN) architectures by automatically exploring multiple architectural parameters of potential network candidates. In most HW-NAS approaches, the HW parameter search space is limited. Hence, HW awareness is tied to only a few degrees of design freedom, leading to the following sub-optimalities - First, it restricts exploration of HW parameters, which can potentially lead to better network candidates; Second, HW-NAS is still entirely a software-centric process where HW-awareness is taken care by an HW function exposed to the NAS process and is oblivious to the actual deployment. To tackle the above challenges, this paper proposes a Co-Design Neural Architecture Search (Co-NAS) approach that simultaneously explores hardware and neural architecture variations, thus allowing for full system optimization. By connecting the mutual impact of variable neural networks and HW parameters on the network's prediction accuracy and on-device efficiency in a shared optimization loop, Co-Nas finds designs of optimum performance and enables HW/SW Co-Design. This work aims to enable more diverse HW search spaces (higher degrees of design freedom) for ML accelerators (such as using a virtual prototype) and efficient exploration by integrating abstract ML accelerator and NN architecture modeling into a comprehensive Co-Nas environment. In our experiments, we explore hardware variations of a baseline accelerator architecture to demonstrate how our work can help find designs with better hardware latency and comparable network accuracy. Designs yielded by our framework provide a speedup of$1.4\times$compared to the baseline on a restricted SW search space at the same HW resources.

Co-optimization of hardware and software in modern deep neural network (DNN) systems can be performed using design space exploration (DSE) tools. Leveraging estimation models to predict design decisions' impact on the system's performance allows a fast evaluation of up to billions of architectural choices. In this work, we propose SimPyler, an end-to-end framework for latency estimations of DNN workloads on machine learning (ML) accelerators. SimPyler represents DNN kernels as graphs executed on abstract accelerator models to simulate the system's latency. By generating the entire simulation infrastructure automated from the DNN operator description, the framework can flexibly adjust to changes at the hardware or algorithmic level, enabling the usage in DSE applications. A key enabler in this automation process is a machine-learning compiler. The framework is implemented in Python, using only open-source software. We demonstrate and validate the proposed methodology by modeling different single-core and multi-core hardware architectures and DNNs, comparable to state-of-the-art. Our experiments show we can estimate the end-to-end latency with an average error of 4.12%

The success of Deep Artificial Neural Networks (DNNs) in many domains created a rich body of research concerned with hardware accelerators for compute-intensive DNN operators. However, implementing such operators efficiently with complex hardware intrinsics such as matrix multiply is a task not yet automated gracefully. Solving this task often requires joint program and data layout transformations. First solutions to this problem have been proposed, such as TVM, UNIT, or ISAMIR, which work on a loop-level representation of operators and specify data layout and possible program transformations before the embedding into the operator is performed. This top-down approach creates a tension between exploration range and search space complexity, especially when also exploring data layout transformations such as im2col, channel packing, or padding. In this work, we propose a new approach to this problem. We created a bottom-up method that allows the joint transformation of both computation and data layout based on the found embedding. By formulating the embedding as a constraint satisfaction problem over the scalar dataflow, every possible embedding solution is contained in the search space. Adding additional constraints and optimization targets to the solver generates the subset of preferable solutions. An evaluation using the VTA hardware accelerator with the Baidu DeepBench inference benchmark shows that our approach can automatically generate code competitive to reference implementations. Further, we show that dynamically determining the data layout based on intrinsic and workload is beneficial for hardware utilization and performance. In cases where the reference implementation has low hardware utilization due to its fixed deployment strategy, we achieve a geomean speedup of up to × 2.813, while individual operators can improve as much as × 170.