Increasingly, pre-trained convolutional neural networks (CNNs) are being deployed for inference in various computer vision applications, both on the server-side in the data centers and at the edge. CNN inference is a very compute-intensive task. It is a challenge to meet performance metrics such as latency and throughput while optimizing power. Special-purpose ASICs and FPGAs are suitable candidates to meet these power and performance budgets simultaneously. Rapidly evolving CNN architectures involve novel convolution operations such as point convolutions, depth separable convolutions, and so on. This leads to substantial variation in the computational structure across CNNs and layers within a CNN. Because of this, FPGA reconfigurability provides an attractive tradeoff compared to ASICs. FPGA-based hardware designers address the structural variability issue by generating a network-specific accelerator for a single network or a class of networks. However, homogeneous accelerators are network agnostic and often sacrifice throughput and FPGA LUTs for flexibility. In this article, we propose an FPGA overlay for efficient processing of CNNs that can be scaled based on the available compute and memory resources of the FPGA. The overlay is configured on the fly through control words sent by the host on a per-layer basis. Unlike current overlays, our architecture exploits all forms of parallelism inside a convolution operation. A constraint system is employed at the host end to find out the per-layer configuration of the overlay that uses all forms of parallelism in the processing of the layer, resulting in the highest throughput for that layer. We studied the effectiveness of our overlay by using it to process AlexNet, VGG16, YOLO, MobileNet, and ResNet-50 CNNs targeting a Virtex7 and a bigger Ultrascale+VU9P FPGAs. The chosen CNNs have a mix of different types of convolution layers and filter sizes, presenting a good variation in model size and structure. Our accelerator reported a maximum throughput of 1,200 GOps/second on the Virtex7, an improvement of 1.2 \( \times \) to 5 \( \times \) over the recent designs. Also, the reported performance density, measured in giga operations per second per KLUT, is 1.3 \( \times \) to 4 \( \times \) improvement over existing works. Similar speed-up and performance density is also observed for the Ultrascale+VU9P FPGA.

Motivated by the increasing interest in the posit numeric format, in this paper we evaluate the accuracy and efficiency of posit arithmetic in contrast to the traditional IEEE 754 32-bit floating-point (FP32) arithmetic. We first design and implement a Posit Arithmetic Unit (PAU), called POSAR, with flexible bit-sized arithmetic suitable for applications that can trade accuracy for savings in chip area. Next, we analyze the accuracy and efficiency of POSAR with a series of benchmarks including mathematical computations, ML kernels, NAS Parallel Benchmarks (NPB), and Cifar-10 CNN. This analysis is done on our implementation of POSAR integrated into a RISC-V Rocket Chip core in comparison with the IEEE 754-based Floting Point Unit (FPU) of Rocket Chip. Our analysis shows that POSAR can outperform the FPU, but the results are not spectacular. For NPB, 32-bit posit achieves better accuracy than FP32 and improves the execution by up to 2%. However, POSAR with 32-bit posit needs 30% more FPGA resources compared to the FPU. For classic ML algorithms, we find that 8-bit posits are not suitable to replace FP32 because they exhibit low accuracy leading to wrong results. Instead, 16-bit posit offers the best option in terms of accuracy and efficiency. For example, 16-bit posit achieves the same Top-1 accuracy as FP32 on a Cifar-10 CNN with a speedup of 18%.

Applications in the Internet of Video Things (IoVT) domain have very tight constraints with respect to power and area. While neuromorphic vision sensors (NVS) may offer advantages over traditional imagers in this domain, the existing NVS systems either do not meet the power constraints or have not demonstrated end-to-end system performance. To address this, we improve on a recently proposed hybrid event-frame approach by using morphological image processing algorithms for region proposal and address the low-power requirement for object detection and classification by exploring various convolutional neural network (CNN) architectures. Specifically, we compare the results obtained from our object detection framework against the state-of-the-art low-power NVS surveillance system and show an improved accuracy of 82.16% from 63.1%. Moreover, we show that using multiple bits does not improve accuracy, and thus, system designers can save power and area by using only single bit event polarity information. In addition, we explore the CNN architecture space for object classification and show useful insights to trade-off accuracy for lower power using lesser memory and arithmetic operations.

Big-data application processing is increasingly geo-distributed, a paradigm shift from the traditional cluster-based processing frameworks. As the communication time for data movement across geo-distributed data centers is not a design criterion for traditional cluster-based processing approaches, there are research gaps in the algorithms used for staging and scheduling big-data applications for geo-distributed clusters. We address these gaps by proposing Harmony, an approach consisting of both staging and scheduling strategies to minimize an application’s total execution time. The staging strategy of Harmony exploits the intra-stage parallelism by having concurrent operators within a stage in contrast to the traditional Apache spark which uses fine-grained stages, thus reducing the computation time within each stage. Secondly, the scheduling strategy of Harmony reduces data transfers between geo-distributed data centers by exploiting data locality and thus reducing communication time and total execution time. The proposed approach Harmony achieves a speedup of two times with respect to geo-distributed Apache Spark. In addition, Harmony achieves a speedup of 1.6 times and 2.1 times when compared with the state-of-the-art framework Iridium for geo-distributed analytics over five locations with uniform and non-uniform network link bandwidths respectively.

Recent advancements in big data processing and machine learning, among others, increase the resource demand for running applications with larger problem sizes. Elastic cloud computing resources with pay-per-use pricing offers new opportunities where large application execution is constrained only by the cost budget. Given a cost budget and a time deadline, this paper introduces a measurement-driven analytical modeling approach to determine the largest Pareto-optimal problem size and its corresponding cloud configuration for execution. We evaluate our approach with a set of representative applications that exhibit a range of resource demand growth patterns on Amazon AWS cloud. We show the existence of cost-time-size Pareto-frontier with multiple sweet spots meeting user constraints. To characterize the cost-performance of cloud resources, we use Performance Cost Ratio (PCR) metric. We extend Gustafson's fixed-time scaling in the context of cloud, and, investigate fixed-cost-time scaling of applications and show that using resources with higher PCR yields better cost-time performance. We discuss a number of useful insights on the trade-off between the execution time and the largest Pareto-optimal problem size, and, show that time deadline could be tightened for a proportionately much smaller reduction of problem size.

Energy efficiency is an area of growing importance in datacenters. While the energy proportionality wall poses a challenge to further improve the dynamic power range of servers, heterogeneous systems offer a new opportunity to achieve higher energy efficiency by improving the match between application workload demands on the large heterogeneous system configuration space. This paper proposes a model-driven energy proportionality analysis of heterogeneous clusters consisting of server nodes with different performance-to-power ratio (PPR). Our analysis shows that inter-node heterogeneity has a positive effect of scaling the energy proportionality wall by exposing configurations with sub-linear energy proportionality. Secondly, analysis of these sub-linear configurations on the 95th percentile response time shows that heterogeneity is beneficial in workloads where the PPR of wimpy nodes is higher than brawny nodes.

Many Integrated Core (MIC) architecture systems are becoming increasingly popular for HPC applications as they have the dual-advantage of accelerating vector processing and a general-purpose programming model. One of the key challenges for energy-efficient execution on MIC architecture systems is to determine time and energy-efficient configurations among a large system configuration space. Given a parallel program with an execution time deadline and an energy budget, we propose a measurement-based analytical modeling approach to determine these system configurations. In contrast to current approaches, we model both inter-and intra-core resource overlaps, memory contention among threads within a core and memory contention across multiple cores. The model is validated against direct measurement using six representative HPC applications on Intel Xeon Phi system. We show that a Pareto frontier consisting of optimal configurations exist for a parallel program executing on MIC architecture systems. To further understand the Pareto frontier, we use the performance-to-power ratio metric (PPR), that quantifies the amount of useful computations performed per unit energy during the execution. The proposed approach can be used to determine what thread affinity is suitable for offloading execution to accelerators such as Xeon Phi and save energy.

Hybrid programming model is becoming increasingly popular for HPC applications as it has the dual-advantage of exploiting inter-node distributed-memory scalability and intra-node shared-memory performance in a cluster system. One of the key challenges for energy efficient execution of hybrid programs is to determine time and energy efficient hardware configurations among a large system configuration space. Given a hybrid program with an execution time deadline and an energy budget, we propose a measurement-based analytical modelling approach to determine these system configurations. In contrast to current approaches, we model both inter and intra-node resource overlaps, memory contention among cores within a node and network contention across multiple nodes. The model invalidated against direct measurement using five representative HPC applications on Intel Xeon and ARM clusters having diverse time-energy performance. We show that a Pareto frontier consisting of optimal configurations exist for a hybrid program running on homogeneous clusters. To further optimize the Pareto frontier, we introduce a new metric, useful computation ratio (UCR) to quantify the degree of resource contentions and communication overheads in an execution. We discuss how UCR and Pareto-optimal configurations can be used in conjunction by system's designers to gain further insights into system resource imbalances, and how application developers can further fine-tune their hybrid programs.

Traditional datacenter systems advocate the use of high-performance hardware, resulting in increased power consumption and cooling costs. With increasing availability of systems having diverse performance-to-power ratios, we analyze the energy efficiency of mixing high-performance and low-power nodes in a cluster. Using a model-driven analysis, we predict the heterogeneous mix of nodes that is the most energy-efficient while maintaining a given deadline. Considering service demands of the workloads on cores, memory and I/O devices, we derive Pareto-optimal configurations by matching the execution rate of different nodes. Our mix and match approach determines heterogeneous configurations that exhibit a "sweet region", where energy usage reduces linearly as the deadline is relaxed. Our analysis shows that mixing high-performance and low-power nodes is more energy-efficient than homogeneous datacenter clusters.