Finetuning language models on a collection of datasets phrased as instructions has been shown to improve model performance and generalization to unseen tasks. In this paper we explore instruction finetuning with a particular focus on (1) scaling the number of tasks, (2) scaling the model size, and (3) finetuning on chain-of-thought data. We find that instruction finetuning with the above aspects dramatically improves performance on a variety of model classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT), and evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation, RealToxicityPrompts). For instance, Flan-PaLM 540B instruction-finetuned on 1.8K tasks outperforms PaLM 540B by a large margin (+9.4% on average). Flan-PaLM 540B achieves state-of-the-art performance on several benchmarks (at time of release), such as 75.2% on five-shot MMLU. We also publicly release Flan-T5 checkpoints,1 which achieve strong few-shot performance even compared to much larger models, such as PaLM 62B. Overall, instruction finetuning is a general method for improving the performance and usability of pretrained language models.

Yi Tay, Jason Wei, Hyung Chung, Vinh Tran, David So, Siamak Shakeri, Xavier Garcia, Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, Denny Zhou, Donald Metzler, Slav Petrov, Neil Houlsby, Quoc Le, Mostafa Dehghani. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

Large language models excel at a wide range of complex tasks. However, enabling general inference in the real world, e.g. for robotics problems, raises the challenge of grounding. We propose embodied language models to directly incorporate real-world continuous sensor modalities into language models and thereby establish the link between words and percepts. Input to our embodied language model are multimodal sentences that interleave visual, continuous state estimation, and textual input encodings. We train these encodings end-to-end, in conjunction with a pre-trained large language model, for multiple embodied tasks including sequential robotic manipulation planning, visual question answering, and captioning. Our evaluations show that PaLM-E, a single large embodied multimodal model, can address a variety of embodied reasoning tasks, from a variety of observation modalities, on multiple embodiments, and further, exhibits positive transfer: the model benefits from diverse joint training across internet-scale language, vision, and visual-language domains. Our largest model with 562B parameters, in addition to being trained on robotics tasks, is a visual-language generalist with state-of-the-art performance on OK-VQA, and retains generalist language capabilities with increasing scale.

Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning, which drastically reduces the number of task-specific training examples needed to adapt the model to a particular application. To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model (PaLM). We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML system which enables highly efficient training across multiple TPU Pods. We demonstrate continued benefits of scaling by achieving state-of-the-art few-shot learning results on hundreds of language understanding and generation benchmarks. On a number of these tasks, PaLM 540B achieves breakthrough performance, outperforming the finetuned state-of-the-art on a suite of multi-step reasoning tasks, and outperforming average human performance on the recently released BIG-bench benchmark. A significant number of BIG-bench tasks showed discontinuous improvements from model scale, meaning that performance steeply increased as we scaled to our largest model. PaLM also has strong capabilities in multilingual tasks and source code generation, which we demonstrate on a wide array of benchmarks. We additionally provide a comprehensive analysis on bias and toxicity, and study the extent of training data memorization with respect to model scale. Finally, we discuss the ethical considerations related to large language models and discuss potential mitigation strategies.

Scaling up training datasets and model parameters have benefited neural network-based language models, but also present challenges like distributed compute, input data bottlenecks and reproducibility of results. We introduce two simple and scalable software libraries that simplify these issues: t5x enables training large language models at scale, while seqio enables reproducible input and evaluation pipelines. These open-source libraries have been used to train models with hundreds of billions of parameters on multi-terabyte datasets. Configurations and instructions for T5-like and GPT-like models are also provided. The libraries can be found at https://github.com/google-research/t5x and https://github.com/google/seqio.

The Mixture-of-Experts (MoE) architecture is showing promising results in improving parameter sharing in multi-task learning (MTL) and in scaling high-capacity neural networks. State-of-the-art MoE models use a trainable "sparse gate'" to select a subset of the experts for each input example. While conceptually appealing, existing sparse gates, such as Top-k, are not smooth. The lack of smoothness can lead to convergence and statistical performance issues when training with gradient-based methods. In this paper, we develop DSelect-k: a continuously differentiable and sparse gate for MoE, based on a novel binary encoding formulation. The gate can be trained using first-order methods, such as stochastic gradient descent, and offers explicit control over the number of experts to select. We demonstrate the effectiveness of DSelect-k on both synthetic and real MTL datasets with up to 128 tasks. Our experiments indicate that DSelect-k can achieve statistically significant improvements in prediction and expert selection over popular MoE gates. Notably, on a real-world, large-scale recommender system, DSelect-k achieves over 22% improvement in predictive performance compared to Top-k. We provide an open-source implementation of DSelect-k.

Large Transformer models yield impressive results on many tasks, but are expensive to train, or even fine-tune, and so slow at decoding that their use and study becomes out of reach. We address this problem by leveraging sparsity. We study sparse variants for all layers in the Transformer and propose Scaling Transformers, a family of next generation Transformer models that use sparse layers to scale efficiently and perform unbatched decoding much faster than the standard Transformer as we scale up the model size. Surprisingly, the sparse layers are enough to obtain the same perplexity as the standard Transformer with the same number of parameters. We also integrate with prior sparsity approaches to attention and enable fast inference on long sequences even with limited memory. This results in performance competitive to the state-of-the-art on long text summarization.

Video streaming is crucial for AI applications that gather videos from sources to servers for inference by deep neural nets (DNNs). Unlike traditional video streaming that optimizes visual quality, this new type of video streaming permits aggressive compression/pruning of pixels not relevant to achieving high DNN inference accuracy. However, much of this potential is left unrealized, because current video streaming protocols are driven by the video source (camera) where the compute is rather limited. We advocate that the video streaming protocol should be driven by real-time feedback from the server-side DNN. Our insight is two-fold: (1) server-side DNN has more context about the pixels that maximize its inference accuracy; and (2) the DNN's output contains rich information useful to guide video streaming. We present DDS (DNN-Driven Streaming), a concrete design of this approach. DDS continuously sends a low-quality video stream to the server; the server runs the DNN to determine where to re-send with higher quality to increase the inference accuracy. We find that compared to several recent baselines on multiple video genres and vision tasks, DDS maintains higher accuracy while reducing bandwidth usage by upto 59% or improves accuracy by upto 9% with no additional bandwidth usage.

Classic query optimization techniques, including predicate pushdown, are of limited use for machine learning inference queries, because the user-defined functions (UDFs) which extract relational columns from unstructured inputs are often very expensive; query predicates will remain stuck behind these UDFs if they happen to require relational columns that are generated by the UDFs. In this work, we demonstrate constructing and applying probabilistic predicates to filter data blobs that do not satisfy the query predicate; such filtering is parametrized to different target accuracies. Furthermore, to support complex predicates and to avoid per-query training, we augment a cost-based query optimizer to choose plans with appropriate combinations of simpler probabilistic predicates. Experiments with several machine learning workloads on a big-data cluster show that query processing improves by as much as 10x.

Knowledge about active radio transmitters is critical for multiple applications: spectrum regulators can use this information to assign spectrum, licensees can identify spectrum usage patterns and provision their future needs, and dynamic spectrum access applications can efficiently pick operating frequency. To achieve these goals, we need a system that continuously senses and characterizes the radio spectrum. Current measurement systems, however, do not scale over time, frequency and space and cannot perform transmitter detection. We address these challenges with theSpectrum Observatory, an end-to-end system for spectrum measurement and characterization. This paper details the design and integration of the Spectrum Observatory, and describes and evaluates the first unsupervised method for detailed characterization of arbitrary transmitters calledTxMiner. We evaluate TxMiner on real-world spectrum measurements collected by the Spectrum Observatory between 30 MHz and 6 GHz and show that it identifies transmitters robustly. Furthermore, we demonstrate the Spectrum Observatory’s capabilities to map the number of active transmitters and their frequency and temporal characteristics, to detect rogue transmitters, and identify opportunities for dynamic spectrum access.

In this paper, we investigate the aerial wireless channel, where a moving drone is deployed to stream content to a set of mobile clients on the ground over a small cell size. Experimental traces collected over more than twenty flights with multiple clients suggest that drone mobility in lateral or vertical path leads to time-selective and frequency-selective wireless channel for a low-altitude drone. The resulting aerial wireless channel can be predicted reasonably well when we model the channel based on the constructive and destructive interference patterns between the line-of-sight path and other propagation paths via nearby reflectors. We propose a novel channel prediction approach to predict the subcarrier SNRs for all clients as drone moves and a novel scheduling approach to select the subset of clients that maximize the network utility using the predicted SNRs. We have implemented the proposed approach on a commodity 802.11n chipset and evaluated in the field over twenty flights, each serving up to 17 live clients. Experiments demonstrate, for the first time, the feasibility of tracking and predicting the aerial Wi-Fi channel, resulting in up to a 56% increase in overall throughput as compared to the conventional 802.11n hotspot, while maintaining fairness across clients.

A network of drone cameras can be deployed to cover live events, such as high-action sports game played on a large field, but managing networked drone cameras in real-time is challenging. Distributed approaches yield suboptimal solutions from lack of coordination but coordination with a centralized controller incurs round-trip latencies of several hundreds of milliseconds over a wireless channel. We propose a fog-networking based system architecture to automatically coordinate a network of drones equipped with cameras to capture and broadcast the dynamically changing scenes of interest in a sports game. We design both optimal and practical algorithms to balance the tradeoff between two metrics: coverage of the most important scenes and streamed video bitrate. To compensate for network round-trip latencies, the centralized controller uses a predictive approach to predict which locations the drones should cover next. The controller maximizes video bitrate by associating each drone to an optimally matched server and dynamically re-assigns drones as relay nodes to boost the throughput in low-throughput scenarios. This dynamic assignment at centralized controller occurs at slower time-scale permitted by round-trip latencies, while the predictive approach and drones' local decision ensures that the system works in real-time. Experimental results over tens of flights on the field suggest our system can achieve really good performance, for example, 8 drones can achieve a tradeoff of 94% coverage and (on average) 2K video support at 20 Mbps by optimizing between coverage and throughput. By dynamically allocating drones to cover the game or act as relays, our system also demonstrates a 2x gain over systems maximizing static coverage alone that achieves only 9 Mbps video throughput.

Driver assistance systems, that rely on vehicular sensors such as cameras, LIDAR and other on-board diagnostic sensors, have progressed rapidly in recent years to increase road safety. Road conditions in developing countries like India are chaotic where roads are not well maintained and thus vehicular sensors alone do not suffice in detecting impending collisions. In this paper, we investigate a collaborative driver assistance system "DRIZY: DRIve eaSY" for such scenarios where inference is drawn from on-board camera feed to alert drivers of obstacles ahead and the cloud uses GPS sensor data uploaded by all vehicles to alert drivers of vehicles in potential collision trajectory. Thus, we combine computer vision and vehicle-to-cloud communication to create comprehensive situational awareness. We prototype our system to consider two types of collisions: vehicle-to-vehicle collisions based on uploading GPS sensor data of vehicles to cloud and vehicle-to-pedestrian collisions based on detecting pedestrians from vehicle's dashboard camera feed. Sensor data processing in each vehicle occurs on smartphone for GPS values which are then uploaded to cloud and on raspberry pi3 for video feeds to make a cost-effective solution. Experiments over both 4G and wireless networks in India show that collaborative driver assistance is feasible in low traffic density within acceptable driver reaction time of <5 sec, but can be limited by the time to process compute-intensive video feeds in real-time. We investigate novel ways to optimize the processing to find an acceptable trade-off.

By placing processing-capable devices at the edge of local wireless access networks, Edge Computing architectures have been recently proposed to connect mobile devices to computational power through a one-hop low-latency wireless link. In this paper, we propose a new design where edge assistance is used to control local data filtering at the mobile devices in bandwidth and energy constrained systems. We focus on real-time monitoring applications, where the video input from mobile devices is processed to centrally detect and recognize objects. The edge processor controls the activation and deactivation of local classifiers implemented by the mobile devices to remove useless portions of video frames. The objective is to adapt the video stream to time-varying bandwidth constraints, while minimizing the additional energy consumption introduced by local processing. To this end, an optimization problem is formulated for a loss function embodying the balance between the risk of violating the available bandwidth and the cost of overly-conservative data filtering. The edge assists the local decision by extracting parameters of the video, such as density of objects of interest in a frame, which influence the output of the sensor. Numerical results, obtained by performing a measurement campaign based on a real implementation, illustrate the tension between energy and bandwidth use for a Haar-feature based cascade classifier.

Camera deployments are ubiquitous, but existing methods to analyze video feeds do not scale and are error-prone. We describe Optasia, a dataflow system that employs relational query optimization to efficiently process queries on video feeds from many cameras. Key gains of Optasia result from modularizing vision pipelines in such a manner that relational query optimization can be applied. Specifically, Optasia can (i) de-duplicate the work of common modules, (ii) auto-parallelize the query plans based on the video input size, number of cameras and operation complexity, (iii) offers chunk-level parallelism that allows multiple tasks to process the feed of a single camera. Evaluation on traffic videos from a large city on complex vision queries shows high accuracy with many fold improvements in query completion time and resource usage relative to existing systems.

Games have transformed from PC-based games to more immersive controllers, but gameplay has been limited to a digital screen or a virtual reality headset. In this demo, we present a novel indoor game that allows a player to maneuver the drone altitude by sensing their emotional state using affective sensors. This game demo introduces two innovations in sensing and actuation for next-generation indoor gaming: maneuvering physical objects such as drones for gameplay and sensing the player's emotional state using affective sensors as a control input. Therefore, the immersive experience of players is heightened same as that of real world. We begin to address the systems challenges in building the gaming platform for "Game-of-drones".

Internet-enabled cameras pervade daily life, generating a huge amount of data, but most of the video they generate is transmitted over wires and analyzed offline with a human in the loop. The ubiquity of cameras limits the amount of video that can be sent to the cloud, especially on wireless networks where capacity is at a premium. In this paper, we present Vigil, a real-time distributed wireless surveillance system that leverages edge computing to support real-time tracking and surveillance in enterprise campuses, retail stores, and across smart cities. Vigil intelligently partitions video processing between edge computing nodes co-located with cameras and the cloud to save wireless capacity, which can then be dedicated to Wi-Fi hotspots, offsetting their cost. Novel video frame prioritization and traffic scheduling algorithms further optimize Vigil's bandwidth utilization. We have deployed Vigil across three sites in both whitespace and Wi-Fi networks. Depending on the level of activity in the scene, experimental results show that Vigil allows a video surveillance system to support a geographical area of coverage between five and 200 times greater than an approach that simply streams video over the wireless network. For a fixed region of coverage and bandwidth, Vigil outperforms the default equal throughput allocation strategy of Wi-Fi by delivering up to 25% more objects relevant to a user's query.

Base-station (BS) cooperation in wireless cellular networks offers a promising approach for interference mitigation. However, the implementation of practical network multi-input multi-output (MIMO) system also faces the challenge of high capacity cost for sharing the user data over the backhaul connections. This paper considers a downlink multi-cell orthogonal frequency-division multiple-access (OFDMA) network where the capacities of the backhaul links between the BSs are limited, and extends the single-antenna BS multi-cell system model considered in our previous work to the multiple-antenna BS case. The BSs use zero-forcing precoding to spatially multiplex multiple users within each cell and to pre-subtract the interference from cooperating BSs that share user data with them. An iterative algorithm that maximizes the downlink network utility is proposed. The algorithm iteratively selects the cooperation links, schedules the users, and optimizes the precoding coefficients and the power spectra for each frequency tone. Numerical results suggest that the use of dynamic cooperation link selection can provide a better trade-off between the downlink sum-rate gain and the backhaul capacity than the earlier fixed link-selection algorithm.

This papers considers the downlink of a wireless multicell downlink orthogonal frequency division multiple-access (OFDMA) system where neighboring base-stations (BSs) can jointly encode and transmit data signals to their users using zero-forcing (ZF) precoding in each frequency tone, but only a limited backhaul-capacity is available for each BS to share its user's data streams with the neighboring BSs. A numerical algorithm is proposed to maximize the network-wide utility of this system subject to backhaul capacity constraints. The proposed algorithm first selects a subset of frequency tones for each BS to share its user's data streams with the neighboring BSs and then, jointly schedules users and adapts the precoding coefficients and the power spectra of the BSs to effectively utilize the limited backhaul capacity. Numerical results show that using the proposed algorithm, the gain in downlink sum-rate per cell can be made to scale linearly with the available backhaul capacity per BS between the no-cooperation and the full-cooperation scenarios.

As field trials and initial deployments of Vectored DSL systems materialize in the next few years, Vectored DSL systems will often co-exist with non-vectored DSL systems sharing the same cable. For such mixed-binder scenarios with vectored and non-vectored lines, this paper presents an optimal Dynamic Spectrum Management (DSM) technique referred to as Mixed-binder Optimum Spectrum Balancing (MixOSB) to find the optimal transmit PSD spectra. When the mixed-binders have short vectored lines and long non-vectored lines, simulations show that MixOSB has substantial performance gains over applying existing DSM techniques independently to the vectored DSL systems and non-vectored DSL systems because it reduces the interference induced by the vectored lines on the non-vectored lines. The high computational complexity of MixOSB prohibits its practical implementation. So, a low-complexity practically-implementable mixed-binder multi-level water-filling (MixMLWF) algorithm is proposed and simulations show its performance is close to the solution given by MixOSB.

In typical home power-line communication (PLC) networks using contention-based access methods, providing quality-of-service (QoS) to high-priority users often comes at the expense of reducing the throughput of low-priority users. This paper proposes a cross-layer protocol which involves interaction between the physical (PHY) and the medium-access-control (MAC) layers, for ensuring politeness of the high-priority users toward the low-priority users for uplink transmission in home PLC networks. This protocol modifies the contention-based CSMA protocol of the MAC layer to exploit the cyclostationarity of the noise in home PLC networks. The PLC noise spectrum has been shown in literature to be periodic with the period of AC line cycle. Using this periodicity, the proposed protocol allows longer medium-access times for low-priority users in every AC line cycle, while meeting the high throughput requirements of the high-priority users. The proposed cross-layer protocol, termed opportunistic CSMA, improves the throughput of the low-priority users by as much as 300% compared to the current CSMA protocols in home PLC networks.

This paper investigates orthogonal frequency-division multiple-access (OFDMA) resource-allocation schemes for two-hop relays in a home powerline communication (PLC) network. Unlike wireless channels, which can have noticeable path loss and shadow effects to bring a substantial relay gain, the powerline channel usually observes a quality of the relay channel on par with the direct link so that simple fixed relay configurations often lose performance instead of gaining any. Besides, the quasi-cyclostationary powerline channel benefits little from diversity combining, a scheme often found in cooperative wireless relay protocols to obtain performance improvements on fast fading channels. To address these special and challenging features of the powerline channel, home-PLC relay protocols that jointly allocate subchannels and power to the source and relay nodes are proposed. Simulation results show that significant improvements of data rates can be achieved by jointly optimizing the subchannel and power allocation of home-PLC relay networks.

In this paper, we present three new detection schemes for the nonorthogonal amplify and forward (NAF) protocol in a cooperative diversity system based on a multiuser detection approach, namely, a channel inversion type detector, a maximal-ratio combining (MRC) type detector, and a biased maximum likelihood (ML) detector. The decision variables in these schemes are linear functionals of the transmitted symbol vector and therefore they are termed as ldquolinearrdquo detectors. A cooperative relay network using the NAF protocol with N - 1 relays is considered where the source transmits symbols in N - 1 blocks of two time slots in a Rayleigh fading environment. For the case of a general linear space-time block coding scheme using K symbols, we present the performance of this protocol in terms of the code symbol error probability when the symbols come from an M-ary phase-shift keying constellation. The approach is further extended to the transmission in N - 1 blocks of L time slots. Numerical results show that the performance of the MRC type detector is superior to that of the channel inversion type detector, while the biased ML detector has the best performance. However, the MRC type detector has the lowest complexity while the biased ML detector the highest.

Power-saving benefits of Dynamic Spectrum Management (DSM) result from its increased robustness to channel and noise changes, as in this paper. A proposed intermittent noise model enables evaluation of the performance of DSM algorithms. Polite Level-1 DSM algorithm consumes much less power than non-polite DSL management algorithms in the presence of intermittent noise. Secondly, a proposed robust level- 2 DSM algorithm compares favorably with methods that change the PSDMASK only upon modem initialization or retraining. The proposed robust level-2 DSM algorithm significantly reduces power consumption. This DSM reduction of DSL transmit power can be even more significant when considering the additional loss of robustness and stability of non-DSM methods.