We introduce Lumos, the first end-to-end multimodal question-answering system with text understanding capabilities. At the core of Lumos is a Scene Text Recognition (STR) component that extracts text from first person point-of-view images, the output of which is used to augment input to a Multimodal Large Language Model (MM-LLM). While building Lumos, we encountered numerous challenges related to STR quality, overall latency, and model inference. In this paper, we delve into those challenges, and discuss the system architecture, design choices, and modeling techniques employed to overcome these obstacles. We also provide a comprehensive evaluation for each component, showcasing high quality and efficiency.

Ting-Wei Wu, Changsheng Zhao, Ernie Chang, Yangyang Shi, Pierce Chuang, Vikas Chandra, Biing Juang. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

From wearables to powerful smart devices, modern automatic speech recognition (ASR) models run on a variety of edge devices with different computational budgets. To navigate the Pareto front of model accuracy vs model size, researchers are trapped in a dilemma of optimizing model accuracy by training and fine-tuning models for each individual edge device while keeping the training GPU-hours tractable. In this paper, we propose Omni-sparsity DNN, where a single neural network can be pruned to generate optimized model for a large range of model sizes. We develop training strategies for Omni-sparsity DNN that allows it to find models along the Pareto front of word-error-rate (WER) vs model size while keeping the training GPU-hours to no more than that of training one singular model. We demonstrate the Omni-sparsity DNN with streaming E2E ASR models. Our results show great saving on training time and resources with similar or better accuracy on LibriSpeech compared to individually pruned sparse models: 2%-6.6% better WER on Test-other.

Creating virtual avatars with realistic rendering is one of the most essential and challenging tasks to provide highly immersive virtual reality (VR) experiences. It requires not only sophisticated deep neural network (DNN) based codec avatar decoders to ensure high visual quality and precise motion expression, but also efficient hardware accelerators to guarantee smooth real-time rendering using lightweight edge devices, like untethered VR headsets. Existing hardware accelerators, however, fail to deliver sufficient performance and efficiency targeting such decoders which consist of multi-branch DNNs and require demanding compute and memory resources. To address these problems, we propose an automation framework, called F-CAD (Facebook Codec avatar Accelerator Design), to explore and deliver optimized hardware accelerators for codec avatar decoding. Novel technologies include 1) a new accelerator architecture to efficiently handle multi-branch DNNs; 2) a multi-branch dynamic design space to enable fine-grained architecture configurations; and 3) an efficient architecture search for picking the optimized hardware design based on both application-specific demands and hardware resource constraints. To the best of our knowledge, F-CAD is the first automation tool that supports the whole design flow of hardware acceleration of codec avatar decoders, allowing joint optimization on decoder designs in popular machine learning frameworks and corresponding customized accelerator design with cycle-accurate evaluation. Results show that the accelerators generated by F-CAD can deliver up to 122.1 frames per second (FPS) and 91.6% hardware efficiency when running the latest codec avatar decoder. Compared to the state-of-the-art designs, F-CAD achieves 4.0× and 2.8× higher throughput, 62.5% and 21.2% higher efficiency than DNNBuilder [1] and HybridDNN [2] by targeting the same hardware device.

Deep Neural Networks (DNNs) represent the state-of-the-art in many Artificial Intelligence (AI) tasks involving images, videos, text, and natural language. Their ubiquitous adoption is limited by the high computation and storage requirements of DNNs, especially for energy-constrained inference tasks at the edge using wearable and IoT devices. One promising approach to alleviate the computational challenges is implementing DNNs using low-precision fixed point (<16 bits) representation. However, the quantization error inherent in any Fixed Point (FxP) implementation limits the choice of bit-widths to maintain application-level accuracy. Prior efforts recommend increasing the network size and/or re-training the DNN to minimize loss due to quantization, albeit with limited success.

The combination of growth in compute capabilities and availability of large datasets has led to a re-birth of deep learning. Deep Neural Networks (DNNs) have become state-of-the-art in a variety of machine learning tasks spanning domains across vision, speech, and machine translation. Deep Learning (DL) achieves high accuracy in these tasks at the expense of 100s of ExaOps of computation; posing significant challenges to efficient large-scale deployment in both resource-constrained environments and data centers.

We present a Discrete Cosine Transform (DCT) unit embedded with Error Detection Sequential (EDS) and Dynamic Voltage Scaling (DVS) circuits to speculatively monitor its noncritical datapaths. This monitoring strategy requires no buffer insertions with only minimal modifications to the existing digital design methodology and is therefore applicable for Field-Programmable Gate Array (FPGA) implementations. The proposed design is implemented in an FPGA. The duty cycles of the constraint clock and the actual clock are differentiated to guide the synthesizer to place the EDS circuits with specific timing margin. The proposed design is tested with two classic images and is able to detect timing errors in the noncritical datapaths due to dynamic process, voltage and temperature (PVT) variations. The DVS circuit correspondingly controls a linear voltage regulator to adjust the supply voltage to the Point of First Failure (PoFF). No actual timing errors are generated, primarily because of the unique speculative characteristic of the proposed monitoring strategy. Our proposed design incurs a 0.3% logic element overhead and 3.5% maximum frequency degradation. By lowering the supply voltage by 8.3%, the proposed design saves up to 16.5% energy when operating at the same frequency as a highly optimized baseline DCT implementation.

With a specified Mean Timing Between Failure (MTBF), the metastability resolution time of synchronizers possibly constrains the system performance. To enhance metastability resolution time under single low-voltage supply environments, Voltage-Boosted Synchronizers (VBSs) consisting of a basic minimum-sized Jamb latch and a switched-capacitor-based charge pump are proposed. The capacitor of the charge pump is sized 13 times the area of the Jamb latch. Two powering strategies of the charge pump, namely Metastability-driven VBS (MVBS) and Clock-driven VBS (CVBS), are proposed. For a 1-year MTBF specification, MVBS and CVBS show 2.0-2.7 and 5.1-9.8 times the performance improvement over the basic Jamb latch, respectively, without incurring large power consumption.

This work focuses on a multi-core VLSI implementation of a multiple-input multiple-output (MIMO) detector utilizing a sphere-decoding algorithm. A complex-domain node traversal algorithm that achieves similar performance results as that of an exhaustive-search algorithm where every node is checked and sorted is also described. A 4×4, 64-QAM hard-output detector utilizing this VLSI design occupies 98k gates, and achieves near-ML performance with an average throughput of 1.22 Gb/s and an energy/bit of 23 pJ/b on a nominal 1.2 V supply in a 0.13μm CMOS process. The hard-output design can be further expanded to provide soft-output capability, and achieves an average throughput of 0.65 Gb/s and reaches 10-5 BER at an SNR of 19.7 dB.

A constant delay (CD) logic style is proposed in this paper, targeting at full-custom high-speed applications. The CD characteristic of this logic style regardless of the logic type makes it suitable in implementing complicated logic expressions such as addition. CD logic exhibits a unique characteristic where the output is pre-evaluated before the inputs from the preceding stage is ready. This feature offers performance advantage over static and dynamic domino logic styles in a single-cycle multistage circuit block. Several design considerations including timing window width adjustment and clock distribution are discussed. Using 65-nm general-purpose CMOS technology, the proposed logic demonstrates an average speedup of 94% and 56% over static and dynamic domino logic, respectively, in five different logic gates. Simulation results of 8-bit ripple carry adders show that CD logic is 39% and 23% faster than the static and dynamic-based adders, respectively. CD logic also demonstrates 39% speedup and 64% (22%) energy-delay product (EDP) reduction from static logic at 100% (10%) data activity in 32-bit carry lookahead adders. For 8-bit Wallace tree multiplier, CD logic achieves a similar speedup with at least 50% EDP reduction across all data activities.

In this work, a new design approach in implementing low-energy, high-performance 64-bit adder using dynamic feedthrough logic (DFTL) is introduced and analyzed. Design issues of using DFTL in several logic depth are analyzed in order to achieve the best optimal balance between performance and power consumption. A ldquotiming windowrdquo technique is also proposed to reduce the amount of excessive power dissipation in the DFTL approach. A 64-bit Sklansky carry-merge adder is used as a benchmark comparison between different logic styles including DFTL, CDL, dynamic, and static logic. Simulation results reveal that the proposed work achieves better performance and is more energy efficient than the other logic styles for high performance adder designs.