Despite significant advancements in deep learning-based CSI compression, some key limitations remain unaddressed. Current approaches predominantly treat CSI compression as a source-coding problem, thereby neglecting transmission errors. Conventional separate source and channel coding suffers from the cliff effect, leading to significant deterioration in reconstruction performance under challenging channel conditions. While existing autoencoder-based compression schemes can be readily extended to support joint source-channel coding, they struggle to capture complex channel distributions and exhibit poor scalability with increasing parameter count. To overcome these inherent limitations of autoencoder-based approaches, we propose Residual-Diffusion Joint Source-Channel Coding (RD-JSCC), a novel framework that integrates a lightweight autoencoder with a residual diffusion module to iteratively refine CSI reconstruction. Our flexible decoding strategy balances computational efficiency and performance by dynamically switching between low-complexity autoencoder decoding and sophisticated diffusion-based refinement based on channel conditions. Comprehensive simulations demonstrate that RD-JSCC significantly outperforms existing autoencoder-based approaches in challenging wireless environments. Furthermore, RD-JSCC offers several practical features, including a low-latency 2-step diffusion during inference, support for multiple compression rates with a single model, robustness to fixed-bit quantization, and adaptability to imperfect channel estimation.

Deep neural network (DNN)-based algorithms are emerging as an important tool for many physical and MAC layer functions in future wireless communication systems, including for large multi-antenna channels. However, training such models typically requires a large dataset of high-dimensional channel measurements, which are very difficult and expensive to obtain. This paper introduces a novel method for generating synthetic wireless channel data using diffusion-based models to produce user-specific channels that accurately reflect real-world wireless environments. Our approach employs a conditional denoising diffusion implicit model (cDDIM) framework, effectively capturing the relationship between user location and multi-antenna channel characteristics. We generate synthetic high fidelity channel samples using user positions as conditional inputs, creating larger augmented datasets to overcome measurement scarcity. The utility of this method is demonstrated through its efficacy in training various downstream tasks such as channel compression and beam alignment. Our diffusion-based augmentation approach achieves over a 1-2 dB gain in NMSE for channel compression, and an 11 dB SNR boost in beamforming compared to prior methods, such as noise addition or the use of generative adversarial networks (GANs).

While neural lossy compression techniques have markedly advanced the efficiency of Channel State Information (CSI) compression and reconstruction for feedback in MIMO communications, efficient algorithms for more challenging and practical tasks—such as CSI compression for future channel prediction and reconstruction with relevant side information—remain underexplored, often resulting in suboptimal performance when existing methods are extended to these scenarios. To that end, we propose a novel framework for compression with side information, featuring an encoding process with fixed-rate compression using a trainable codebook for codeword quantization, and a decoding procedure modeled as a backward diffusion process conditioned on both the codeword and the side information. Experimental results show that our method significantly outperforms existing CSI compression algorithms, often yielding over twofold performance improvement by achieving comparable distortion at less than half the data rate of competing methods in certain scenarios. These findings underscore the potential of diffusion-based compression for practical deployment in communication systems.

Attention-based transformers have achieved tremendous success across a variety of disciplines including natural languages. To deepen our understanding of their sequential modeling capabilities, there is a growing interest in using Markov input processes to study them. A key finding is that when trained on first-order Markov chains, transformers with two or more layers consistently develop an induction head mechanism to estimate the in-context bigram conditional distribution. In contrast, single-layer transformers, unable to form an induction head, directly learn the Markov kernel but often face a surprising challenge: they become trapped in local minima representing the unigram distribution, whereas deeper models reliably converge to the ground-truth bigram. While single-layer transformers can theoretically model first-order Markov chains, their empirical failure to learn this simple kernel in practice remains a curious phenomenon. To explain this contrasting behavior of single-layer models, in this paper we introduce a new framework for a principled analysis of transformers via Markov chains. Leveraging our framework, we theoretically characterize the loss landscape of single-layer transformers and show the existence of global minima (bigram) and bad local minima (unigram) contingent on data properties and model architecture. We precisely delineate the regimes under which these local optima occur. Backed by experiments, we demonstrate that our theoretical findings are in congruence with the empirical results. Finally, we outline several open problems in this arena.

This paper investigates the causal rate-distortion function for networked Linear Quadratic Gaussian (LQG) control systems with two encoders and a single decoder, a longstanding open problem in information/control theory. While previous work has explored the causal rate-distortion function for single-encoder and feedback-enabled networked settings, the case of networks without feedback remains unaddressed. We establish a novel directed information lower bound, the first derived for the networked LQG setting. We further demonstrate the optimality of linear, independent encoders and linear decoders for optimizing this lower bound. By reducing the original infinite-dimensional optimization problem to a finite-dimensional one, our approach simplifies the analysis. Additionally, our directed information lower bound provides an alternate proof for the sufficiency of linear encoders in point-to-point settings, both with and without side information, extending prior results in the literature.

Risk-averse modeling is critical in safety-sensitive and high-stakes applications. Conditional Value-at-Risk (CVaR) quantifies such risk by measuring the expected loss in the tail of the loss distribution, and minimizing it provides a principled framework for training robust models. However, direct CVaR minimization remains challenging due to the difficulty of accurately estimating rare, high-loss events—particularly at extreme quantiles. In this work, we propose a novel training framework that synthesizes informative samples for CVaR optimization using score-based generative models. Specifically, we guide a diffusion-based generative model to sample from a reweighted distribution that emphasizes inputs likely to incur high loss under a pretrained reference model. These samples are then incorporated via a loss-weighted importance sampling scheme to reduce noise in stochastic optimization. We establish convergence guarantees and show that the synthesized, high-loss-emphasized dataset substantially contributes to the noise reduction. Empirically, we validate the effectiveness of our approach across multiple settings, including a real-world wireless channel compression task, where our method achieves significant improvements over standard risk minimization strategies.

The design of reliable and efficient codes for channels with feedback remains a longstanding challenge in communication theory. While significant improvements have been achieved by leveraging deep learning techniques, neural codes often suffer from high computational costs, a lack of interpretability, and limited practicality in resource-constrained settings. We focus on designing low-complexity coding schemes that are interpretable and more suitable for communication systems. We advance both analytical and neural codes. First, we demonstrate that PowerBlast, an analytical coding scheme inspired by Schalkwijk-Kailath (SK) and Gallager-Nakiboğlu (GN) schemes, achieves notable reliability improvements over both SK and GN schemes, outperforming neural codes in high signal-to-noise ratio (SNR) regions. Next, to enhance reliability in low-SNR regions, we propose LightCode, a lightweight neural code that achieves state-of-the-art reliability while using a fraction of memory and compute compared to existing deep-learning-based codes. Finally, we systematically analyze the learned codes, establishing connections between LightCodeand PowerBlast, identifying components crucial for performance, and providing interpretation aided by linear regression analysis.

In modern wireless systems, the feedback of DownLink (DL) Channel State Information (CSI) from User Equipment (UE) to Base Stations (BS) may require substantial computational and feedback bandwidth overheads. A promising approach to improve feedback efficiency is to leverage side information which is correlated to DL CSI. Despite potential of doing so, critical aspects remain underexplored in current research, particularly the quantification of the benefits and the inherent limitations of utilizing side information. This paper addresses these gaps by introducing a novel algorithm to compute the rate-distortion function for general compression scenarios incorporating side information. We apply this algorithm to the DL CSI feedback problem having UL CSI as the side information and generate rate-distortion functions. Using the estimated rate-distortion functions, we measure the gain of side information over diverse feedback rates and UE mobility profiles. The results reveal that the benefits of leveraging side information are particularly significant for UEs characterized by high mobility and constrained to operate at low feedback overheads.

In this paper, we consider aK-user interference channel where interference among the users is neither too strong nor too weak, a scenario that is relatively underexplored in the literature. We propose a novel deep learning-based approach to design the encoder and decoder functions that aim to maximize the sumrate of the interference channel for discrete constellations. We first consider the MaxSINR algorithm, a state-of-the-art linear scheme for Gaussian inputs, as the baseline and then propose a modified version of the algorithm for discrete inputs. We then propose a neural network-based approach that learns a non-linear constellation mapping with the objective of maximizing the sumrate. We provide numerical results to show that the constellations learned by the neural network-based approach provide enhanced alignments, not just in beamforming directions but also in terms of the effective constellation at the receiver, thereby leading to improved sum-rate performance.

For millimeter wave (mmWave) communication, fast and accurate beam alignment is essential but challenging. Site-specific beam adaptation using deep learning is a very promising paradigm for beam alignment, but such methods typically require a lot of clean channel measurements for training, which can be difficult or even impossible to achieve in practice. This paper introduces a novel method to learn beam alignment policies using only uplink (UL) pilot measurements. The proposed method integrates a generative adversarial network (GAN)-based channel estimation (CE) model with an unsupervised deep learning model beam alignment engine (BAE). We introduce an efficient form of dataset amplification for improved training that leverages the randomness of the deep generative model (DGM) and an early stopping mechanism. Our experiments show that the GAN-BAE method achieves a better signal-to-noise ratio (SNR) by nearly 3 dB compared to compressed sensing (CS) methods such as orthogonal matching pursuit (OMP) and EM-GM-AMP (an Approximate Message Passing algorithm), especially when there are limited pilot measurements from each mobile user.

Wireless use cases such as spectrum sharing and Massive Machine Type Communications (mMTC) can benefit from the detection of unknown signals, which includes estimating their received power as well as other key characteristics such as bandwidth, modulation type, and waveform. While conventional signal detection methods are susceptible to noise, deep learning (DL) models offer a more robust alternative. Previously, DL models were used for solving simpler problems, focusing mainly on modulation recognition. We propose an advanced DL neural network structure that extracts the parameters of 5G NR frequency range 2 (FR2) mmWave test model waveforms. We evaluate our framework on a state-of-the-art signal generator and vector signal analyzer (VSA) that mimics real-world detection. Our work shows that incorporating curriculum training (CT) on both additive white Gaussian noise (AWGN) and frequency shift error enhances the model's accuracy across all SNR and frequency shift ranges. We further enhance the accuracy by employing the error vector magnitude (EVM) function to prioritize the top five scored parameters and validate selected parameters. As a result, our method consistently achieves an accuracy rate exceeding 90% when extracting the key parameters from 5G NR FR2 mmWave waveforms at diverse noise levels.

Progress in designing channel codes has been driven by human ingenuity and, fittingly, has been sporadic. Polar codes, developed on the foundation of Arikan’s polarization kernel, represent the latest breakthrough in coding theory and have emerged as the state-of-the-art error-correction code for short-to-medium block length regimes. In an effort to automate the invention of good channel codes, especially in this regime, we explore a novel, non-linear generalization of Polar codes, which we call DeepPolar codes. DeepPolar codes extend the conventional Polar coding framework by utilizing a larger kernel size and parameterizing these kernels and matched decoders through neural networks. Our results demonstrate that these data-driven codes effectively leverage the benefits of a larger kernel size, resulting in enhanced reliability when compared to both existing neural codes and conventional Polar codes.

Clustered Federated Learning (CFL) is a promising distributed learning framework that addresses data heterogeneity issues across multiple clients by grouping clients and providing a shared generalized model for each group. However, under privacy-preserving federated learning protocols where there is no direct sharing of clients' local datasets, existing approaches often fail to find optimal client groupings resulting in sub-optimal performance. In this paper, we propose a novel CFL algorithm that achieves robust clustering and learning performance. Conceptually, our algorithm groups clients that exhibit similarity in their model updates by periodically accumulating and clustering the gradients that clients compute for various models. The proposed algorithm is shown to achieve a near-optimal error rate for stochastic convergence to optimal models under mild conditions. We present a detailed analysis of the algorithm along with an evaluation on several CFL benchmarks demonstrating that it outperforms existing approaches in terms of convergence speed, clustering accuracy, and task performance.

Data-parallel SGD is the de facto algorithm for distributed optimization, especially for large scale machine learning. Despite its merits, communication bottleneck is one of its persistent issues. Most compression schemes to alleviate this either assume noiseless communication links, or fail to achieve good performance on practical tasks. In this paper, we close this gap and introduce **LASER**: **L**ine**A**r Compre**S**sion in Wir**E**less Dist**R**ibuted Optimization. LASER capitalizes on the inherent low-rank structure of gradients and transmits them efficiently over the noisy channels. Whilst enjoying theoretical guarantees similar to those of the classical SGD, LASER shows consistent gains over baselines on a variety of practical benchmarks. In particular, it outperforms the state-of-the-art compression schemes on challenging computer vision and GPT language modeling tasks. On the latter, we obtain 50-64% improvement in perplexity over our baselines for noisy channels.

There has been growing interest in computing rate-distortion functions for real-world data, as they can provide a theoretical benchmark for compression problems. However, a generalized form of rate-distortion that includes side information and coding for computing has been underexplored, despite its relevance in modern compression problems. To address this gap, we propose a new method for estimating the rate-distortion function for computing with side information, using a Lagrangian framework with neural network-parametrized encoding and decoding strategies. This approach enables targeting specific points on the rate-distortion curve through gradient-based optimization. Our methodology is validated in synthetic environments where rate-distortion functions are known, ensuring accuracy in estimation. Additionally, we extend its application to practical, high-dimensional channel state information compression scenarios. We provide rate-distortion estimation results on these scenarios, which in turn enables us to quantify the usefulness of side information in the practical scenarios.11The code is available at https://github.com/Heasung-Kimlrate-distortion-side-information.

In steganography, selecting an optimal cover image—referred to as cover selection—is pivotal for effective message concealment. Traditional methods have typically employed exhaustive searches to identify images that conform to specific perceptual or complexity metrics. However, the relationship between these metrics and the actual message hiding efficacy of an image is unclear, often yielding less-than-ideal steganographic outcomes. Inspired by recent advancements in generative models, we introduce a novel cover selection framework, which involves optimizing within the latent space of pretrained generative models to identify the most suitable cover images, distinguishing itself from traditional exhaustive search methods. Our method shows significant advantages in message recovery and image quality. We also conduct an information-theoretic analysis of the generated cover images, revealing that message hiding predominantly occurs in low-variance pixels, reflecting the waterfilling algorithm's principles in parallel Gaussian channels.

In recent years, transformer-based models have revolutionized deep learning, particularly in sequence modeling. To better understand this phenomenon, there is a growing interest in using Markov input processes to study transformers. However, our current understanding in this regard remains limited with many fundamental questions about how transformers learn Markov chains still unanswered. In this paper, we address this by focusing on first-order Markov chains and single-layer transformers, providing a comprehensive characterization of the learning dynamics in this context. Specifically, we prove that transformer parameters trained on next-token prediction loss can either converge to global or local minima, contingent on the initialization and the Markovian data properties, and we characterize the precise conditions under which this occurs. To the best of our knowledge, this is the first result of its kind highlighting the role of initialization. We further demonstrate that our theoretical findings are corroborated by empirical evidence. Based on these insights, we provide guidelines for the initialization of single-layer transformers and demonstrate their effectiveness. Finally, we outline several open problems in this arena. Code is available at: \url{https://github.com/Bond1995/Markov}.

We formalize the problem of prompt compression for large language models (LLMs) and present a framework to unify token-level prompt compression methods which create hard prompts for black-box models. We derive the distortion-rate function for this setup as a linear program, and provide an efficient algorithm to compute this fundamental limit via the dual of the linear program. Using the distortion-rate function as the baseline, we study the performance of existing compression schemes on a synthetic dataset consisting of prompts generated from a Markov chain, natural language queries, and their respective answers. Our empirical analysis demonstrates the criticality of query-aware prompt compression, where the compressor has knowledge of the downstream task/query for the black-box LLM. We show that there is a large gap between the performance of current prompt compression methods and the optimal strategy, and propose Adaptive QuerySelect, a query-aware, variable-rate adaptation of a prior work to close the gap. We extend our experiments to a small natural language dataset to further confirm our findings on our synthetic dataset.

A two-user interference channel is a canonical model for multiple one-to-one communications, where two transmitters wish to communicate with their receivers via a shared medium, examples of which include pairs of base stations and handsets near the cell boundary that suffer from interference. Practical codes and the fundamental limit of communications are unknown for interference channels as mathematical analysis becomes intractable. Hence, simple heuristic coding schemes are used in practice to mitigate interference, e.g., time division, treating interference as noise, and successive interference cancellation. These schemes are nearly optimal for extreme cases: when interference is strong or weak. However, there is no optimality guarantee for channels with moderate interference. Here we combine deep learning and network information theory to overcome the limitation on the tractability of analysis and construct finite-blocklength coding schemes for channels with various interference levels. We show that carefully designed and trained neural codes using network information theoretic insight can achieve several orders of reliability improvement for channels with moderate interference. Furthermore, we present the interpretation of the learned codes based on the codeword distance and the Centered Kernel Alignment (CKA) analysis.

In decoding linear block codes, it was shown that noticeable reliability gains can be achieved by introducing learnable parameters to the Belief Propagation (BP) decoder. Despite the success of these methods, there are two key open problems. The first is the lack of interpretation of the learned weights, and the other is the lack of analysis for non-AWGN channels. In this work, we aim to bridge this gap by providing insights into the weights learned and their connection to the structure of the underlying code. We show that the weights are heavily influenced by the distribution of short cycles in the code. We next look at the performance of these decoders in non-AWGN channels, both synthetic and over-the-air channels, and study the complexity vs. performance trade-offs, demonstrating that increasing the number of parameters helps significantly in complex channels. Finally, we show that the decoders with learned weights achieve higher reliability than those with weights optimized analytically under the Gaussian approximation.

In modern communication systems with feedback, there are increasingly more scenarios where the transmitter has much less power than the receiver (e.g., medical implant devices), which we refer to as noise-asymmetric channels. For such channels, the feedback link is of higher quality than the forward link. However, feedback schemes for cellular communications, such as hybrid ARQ, do not fully utilize the high-quality feedback link. To this end, we introduce Compressed Error Hybrid ARQ, a generalization of hybrid ARQ tailored for noise-asymmetric channels; the receiver sends its estimated message to the transmitter, and the transmitter harmoniously switches between hybrid ARQ and compressed error retransmission. We show that our proposed method significantly improves reliability, latency, and spectral efficiency compared to the conventional hybrid ARQ in various practical scenarios where the transmitter is resource-constrained.

Efficient compression of correlated data is essential to minimize communication overload in multi-sensor networks. In such networks, each sensor independently compresses the data and transmits them to a central node. A decoder at the central node decompresses and passes the data to a pre-trained machine learning-based task model to generate the final output. Due to limited communication bandwidth, it is important for the compressor to learn only the features that are relevant to the task. Additionally, the final performance depends heavily on the total available bandwidth. In practice, it is common to encounter varying availability in bandwidth. Since higher bandwidth results in better performance, it is essential for the compressor to dynamically take advantage of the maximum available bandwidth at any instant. In this work, we propose a novel distributed compression framework composed of independent encoders and a joint decoder, which we call neural distributed principal component analysis (NDPCA). NDPCA flexibly compresses data from multiple sources to any available bandwidth with a single model, reducing compute and storage overhead. NDPCA achieves this by learning low-rank task representations and efficiently distributing bandwidth among sensors, thus providing a graceful trade-off between performance and bandwidth. Experiments show that NDPCA improves the success rate of multi-view robotic arm manipulation by 9% and the accuracy of object detection tasks on satellite imagery by 14% compared to an autoencoder with uniform bandwidth allocation.

The optimal coding scheme for Additive White Gaussian noise (AWGN) channels with noisy output feedback has been unknown for several decades. The best-known linear scheme is by Chance and Love, where the coefficients of the linear scheme are numerically optimized based on unique observations. In this paper, we introduce a new class of linear coding schemes, calledsequential linear schemes, where the encoder sequentially updates a linear state process based on feedback. We then derive the optimal scheme within this class, in a closed form, by formulating a novel Markov decision process and solving it via dynamic programming. We demonstrate that our scheme outperforms the Chance-Love scheme for channels with noisy feedback and coincides with the Shalkwijk-Kailath scheme for channels with noiseless feedback. This problem is an instance of decentralized controlwithout any common informationand, to the best of our knowledge, the first such scenario where we can derive analytical solutions using dynamic programming.

With advances in artificial intelligence technology, Voice-based Conversational Agents (VCAs) can now imitate human abilities, sometimes almost indistinguishably from humans. However, concerns have been raised that too much perceived similarity can trigger threats and fears among users. This raises a question: Should VCAs be able to imitate humans perfectly? To address this, we explored what influences the negative aspects of user experience in human-like VCAs. We conducted a qualitative exploratory study to elicit participants’ perceptions and feelings of human-like VCAs through comparable video prototypes of human–agent conversation and human–human conversation. We discovered that the dialogues of the human-likeness outside of the expressed purpose of a VCA and expressions pretending to come from a human identity could lead to negative experiences with VCAs. Based on our findings, we discussed design directions for overcoming potential issues of human imitation.

We observe that current Deep Learning (DL)-based Channel State Information (CSI) encoder and decoder architectures achieve a distortion which is highly channel-dependent. To exploit this, we propose a novel learning-based variable-rate coding scheme to reduce overheads associated with CSI feedback. To that end, we propose an architecture which combines (a) training an efficient predictor for the distortion rate tradeoffs achievable for a given channel, and (b) optimization of a decision logic which allocates rates based on the predicted distortion. We evaluate our approach on various wireless channel datasets including the 3GPP 3D channel model and COST2100 with Massive MIMO channel model, and show significant potential reductions of up to 20% in the CSI feedback overhead.

A critical aspect of reliable communication involves the design of codes that allow transmissions to be robustly and computationally efficiently decoded under noisy conditions. Advances in the design of reliable codes have been driven by coding theory and have been sporadic. Recently, it is shown that channel codes that are comparable to modern codes can be learned solely via deep learning. In particular, Turbo Autoencoder (TurboAE), introduced by Jiang et al., is shown to achieve the reliability of Turbo codes for Additive White Gaussian Noise channels.In this paper, we focus on applying the idea of TurboAE to various practical channels, such as fading channels and chirp noise channels. We introduce TurboAE-TI, a novel neural architecture that combines TurboAE with a trainable interleaver design. We develop a carefully-designed training procedure and a novel interleaver penalty function that are crucial in learning the interleaver and TurboAE jointly. We demonstrate that TurboAE-TI outperforms TurboAE and LTE Turbo codes for several channels of interest. We also provide interpretation analysis to better understand TurboAE-TI.

In this paper, we introduce a neural-augmented decoder for Turbo codes called TINYTURBO . TINYTURBO has complexity comparable to the classical max-log-MAP algorithm but has much better reliability than the max-log-MAP baseline and performs close to the MAP algorithm. We show that TINYTURBO exhibits strong robustness on a variety of practical channels of interest, such as EPA and EVA channels, which are included in the LTE standards. We also show that TINYTURBO strongly generalizes across different rate, blocklengths, and trellises. We verify the reliability and efficiency of TINYTURBO via over-the-air experiments.

Semiglobal matching is an accurate stereo depth estimation algorithm, whereas implementing the high-throughput architecture has been challenging due to the inherent recursion on inter-pixel cost aggregation. Especially, the computation on horizontal scan pass is the critical path causing the throughput bottleneck. In this paper, we propose a new cluster-wise cost aggregation algorithm and its optimized architecture that enables to pipeline the inter-pixel aggregation and parallelize the scanline-level disparity computation. The proposed approach is performed not on every pixel but on each group of pixels, which significantly alleviates the timing constraint for the recursion. The disparity values at shifted multiple pixel positions are concurrently computed within a single clock period. We also propose the memory reduction scheme selecting a tiny number of informative values, which achieves 96% memory reduction compared to the straightforward approach storing overall values. The system-on-chip-based tiled processing scheme is employed, which allows the implementation without an external memory. The proposed architecture computes a depth map with 128 disparity levels at 103 frames per second on a full HD image on the Zynq ultrascale+ MPSoC platform, thus providing 2.6 times faster performance with a comparable accuracy compared to the state-of-the-art 8-path semiglobal matching implementation.

The two-user interference channel is a model for multi one-to-one communications, where two transmitters wish to communicate with their corresponding receivers via a shared wireless medium. Two most common and simple coding schemes are Time Division (TD) and Treating Interference as Noise (TIN). Interestingly, it is shown that there exists an asymptotic scheme, called Han-Kobayashi scheme, that performs better than TD and TIN. However, Han-Kobayashi scheme has impractically high complexity and is designed for asymptotic settings, which leads to a gap between information theory and practice. In this paper, we focus on designing practical codes for interference channels. As it is challenging to analytically design practical codes with feasible complexity, we apply deep learning to learn codes for interference channels. We demonstrate that DeepIC, a convolutional neural network-based code with an iterative decoder, outperforms TD and TIN by a noticeable margin for two-user Additive White Gaussian Noise channels with moderate amount of interference.

In this demonstration, we present AB9 SoC system, a single-chip solution for AI application. It provides the reconfigurable and programmable architecture to support the general computations for a variety of neural networks. The AB9 SoC is implemented using TSMC 28-nm process technology with a chip size of 17×23 mm2and 1GHz operating frequency.

In this paper, we develop a framework for an autoencoder based transmission strategy for achieving distributed interference alignment and optimal power allocation in a multiuser interference channel. The users in the interference channel have access to the local channel state information only. We compare the explicit schemes, such as MaxSINR [1], against the autoencoder schemes. We find that the MaxSINR schemes outperform the autoencoder networks which are either jointly or distributively trained from scratch. However, we find that the autoencoders which are pretrained with the beamforming vectors and the power allocation obtained from the explicit schemes outperform the explicit schemes when the interference gets stronger. The explicit schemes perform well as they are effective in choosing the set of users which are to be suppressed. The pretrained autoencoders benefit from this initialization, and also from the fact that end to end training can improve their performance even further. We showcase our performance comparison results for 5 user interference channels with different levels of interference.

In this paper, we consider a communication system where a sender sends messages over a memoryless Gaussian point-to-point channel to a receiver and receives the output feedback over another Gaussian channel with known variance and unit delay. The sender sequentially transmits the message over multiple times till a certain error performance is achieved. The aim of our work is to design a transmission strategy to process every transmission with the information that was received in the previous feedback and send a signal so that the estimation error drops as quickly as possible. The optimal code is unknown for channels with noisy output feedback when the block length is finite. Even within the family of linear codes, optimal codes are unknown in general. Bridging this gap, we propose a family of linear sequential codes and provide a dynamic programming (DP) algorithm to solve for a closed form expression for the optimal code within a class of sequential linear codes. The optimal code discovered via DP is a generalized version of which the Schalkwijk-Kailath (SK) scheme is one special case with noiseless feedback; our proposed code coincides with the celebrated SK scheme for noiseless feedback settings.

Neural architecture search (NAS) has been very successful at outperforming human-designed convolutional neural networks (CNN) in accuracy, and when hardware information is present, latency as well. However, NAS-designed CNNs typically have a complicated topology, therefore, it may be difficult to design a custom hardware (HW) accelerator for such CNNs. We automate HW-CNN codesign using NAS by including parameters from both the CNN model and the HW accelerator, and we jointly search for the best model-accelerator pair that boosts accuracy and efficiency. We call this Codesign-NAS. In this paper we focus on defining the Codesign-NAS multiobjective optimization problem, demonstrating its effectiveness, and exploring different ways of navigating the codesign search space. For CIFAR-10 image classification, we enumerate close to 4 billion model-accelerator pairs, and find the Pareto frontier within that large search space. This allows us to evaluate three different reinforcement-learning-based search strategies. Finally, compared to ResNet on its most optimal HW accelerator from within our HW design space, we improve on CIFAR-100 classification accuracy by 1.3% while simultaneously increasing performance/area by 41% in just ~1000 GPU-hours of running Codesign-NAS.

Field-programmable gate arrays (FPGAs) have become a popular compute platform for convolutional neural network (CNN) inference; however, the design of a CNN model and its FPGA accelerator has been inherently sequential. A CNN is first prototyped with no-or-little hardware awareness to attain high accuracy; subsequently, an FPGA accelerator is tuned to that specific CNN to maximize its efficiency. Instead, we formulate a neural architecture search (NAS) optimization problem that contains parameters from both the CNN model and the FPGA accelerator, and we jointly search for the best CNN model-accelerator pair that boosts accuracy and efficiency -we call this Codesign-NAS. In this paper we focus on defining the Codesign-NAS multiobjective optimization problem, demonstrating its effectiveness, and exploring different ways of navigating the codesign search space. For Cifar-10 image classification, we enumerate close to 4 billion model-accelerator pairs, and find the Pareto frontier within that large search space. Next we propose accelerator innovations that improve the entire Pareto frontier. Finally, we compare to ResNet on a highly-tuned accelerator, and show that using codesign, we can improve on Cifar-100 classification accuracy by 1.8% while simultaneously increasing performance/area by 41% in just 1000 GPU-hours of running Codesign-NAS, thus demonstrating that our automated codesign approach is superior to sequential design of a CNN model and accelerator.

Designing channel codes is one of the core research areas for modern communication systems. Canonical channel codes asymptotically achieve near-capacity performance under large block length regime for additive white gaussian noise channels. However, this achieved success does not generalize to many channels. Channels with output feedback, proposed by Shannon, is one of such channels where practical codes have been unknown for several decades.Recently it has been demonstrated that deep learning based code outperforms the state-of-the-art codes for channels with output feedback. While the success is promising and inspiring, there are a few major challenges that need to be addressed. Firstly, the channel assumes a feedback with a unit step delay, which is not very practical. Second is the lack of generalization to larger block lengths. In this work, we propose Feedback Auto Turbo Encoder (FTAE) which harmoniously combines interleaver and iterative decoding with CNN architectures and demonstrate the blocklength gain and improved performance in the block feedback setting.

Convolutional neural networks (CNNs) have recently become the state-of-the-art in a diversity of AI tasks. Despite their popularity, CNN inference still comes at a high computational cost. A growing body of work aims to alleviate this by exploiting the difference in the classification difficulty among samples and early-exiting at different stages of the network. Nevertheless, existing studies on early exiting have primarily focused on the training scheme, without considering the use-case requirements or the deployment platform. This work presents HAPI, a novel methodology for generating high-performance early-exit networks by co-optimising the placement of intermediate exits together with the early-exit strategy at inference time. Furthermore, we propose an efficient design space exploration algorithm which enables the faster traversal of a large number of alternative architectures and generates the highest-performing design, tailored to the use-case requirements and target hardware. Quantitative evaluation shows that our system consistently outperforms alternative search mechanisms and state-of-the-art early-exit schemes across various latency budgets. Moreover, it pushes further the performance of highly optimised hand-crafted early-exit CNNs, delivering up to 5.11× speedup over lightweight models on imposed latency-driven SLAs for embedded devices.

Automatic speech recognition (ASR) via call is essential for various applications, including AI for contact center (AICC) services. Despite the advancement of ASR, however, most publicly available call-based speech corpora such as Switchboard are old-fashioned. Also, most existing call corpora are in English and mainly focus on open domain dialog or general scenarios such as audiobooks. Here we introduce a new large-scale Korean call-based speech corpus under a goal-oriented dialog scenario from more than 11,000 people, i.e., ClovaCall corpus. ClovaCall includes approximately 60,000 pairs of a short sentence and its corresponding spoken utterance in a restaurant reservation domain. We validate the effectiveness of our dataset with intensive experiments using two standard ASR models. Furthermore, we release our ClovaCall dataset and baseline source codes to be available via https://github.com/ClovaAI/ClovaCall. Copyright © 2020 ISCA

Neural architecture search (NAS) enables researchers to automatically explore broad design spaces in order to improve efficiency of neural networks. This efficiency is especially important in the case of on-device deployment, where improvements in accuracy should be balanced out with computational demands of a model. In practice, performance metrics of model are computationally expensive to obtain. Previous work uses a proxy (e.g., number of operations) or a layer-wise measurement of neural network layers to estimate end-to-end hardware performance but the imprecise prediction diminishes the quality of NAS. To address this problem, we propose BRP-NAS, an efficient hardware-aware NAS enabled by an accurate performance predictor-based on graph convolutional network (GCN). What is more, we investigate prediction quality on different metrics and show that sample efficiency of the predictor-based NAS can be improved by considering binary relations of models and an iterative data selection strategy. We show that our proposed method outperforms all prior methods on NAS-Bench-101, NAS-Bench-201 and DARTS. Finally, to raise awareness of the fact that accurate latency estimation is not a trivial task, we release LatBench -- a latency dataset of NAS-Bench-201 models running on a broad range of devices.

Network compression reduces the computational complexity and memory consumption of deep neural networks by reducing the number of parameters. In SVD-based network compression the right rank needs to be decided for every layer of the network. In this paper we propose an efficient method for obtaining the rank configuration of the whole network. Unlike previous methods which consider each layer separately, our method considers the whole network to choose the right rank configuration. We propose novel accuracy metrics to represent the accuracy and complexity relationship for a given neural network. We use these metrics in a non-iterative fashion to obtain the right rank configuration which satisfies the constraints on FLOPs and memory while maintaining sufficient accuracy. Experiments show that our method provides better compromise between accuracy and computational complexity/memory consumption while performing compression at much higher speed. For VGG-16 our network can reduce the FLOPs by 25% and improve accuracy by 0.7% compared to the baseline, while requiring only 3 minutes on a CPU to search for the right rank configuration. Previously, similar results were achieved in 4 hours with 8 GPUs. The proposed method can be used for lossless compression of a neural network as well. The better accuracy and complexity compromise, as well as the extremely fast speed of our method make it suitable for neural network compression.

Designing channel codes under low latency constraints is one of the most demanding requirements in 5G standards. However, sharp characterizations of the performances of traditional codes are only available in the large block lengths limit. Code designs are guided by those asymptotic analyses and require large block lengths and long latency to achieve the desired error rate. Furthermore, when the codes designed for one channel (e.g. Additive White Gaussian Noise (AWGN) channel) are used for another (e.g. non-AWGN channels), heuristics are necessary to achieve any non trivial performance - thereby severely lacking in robustness as well as adaptivity. Obtained by jointly designing recurrent neural network (RNN) based encoder and decoder, we propose an end-to-end learned neural code which outperforms canonical convolutional code under block settings. With this gained experience of designing a novel neural block code, we propose a new class of codes under low latency constraint - Low-latency Efficient Adaptive Robust Neural (LEARN) codes, which outperform the state-of-the-art low latency codes as well as exhibit robustness and adaptivity properties. LEARN codes show the potential of designing new versatile and universal codes for future communications via tools of modern deep learning coupled with communication engineering insights.

Designing codes that combat the noise in a communication medium has remained a significant area of research in information theory as well as wireless communications. Asymptotically optimal channel codes have been developed by mathematicians for communicating under canonical models after over 60 years of research. On the other hand, in many non-canonical channel settings, optimal codes do not exist and the codes designed for canonical models are adapted via heuristics to these channels and are thus not guaranteed to be optimal. In this work, we make significant progress on this problem by designing a fully end-to-end jointly trained neural encoder and decoder, namely, Turbo Autoencoder (TurboAE), with the following contributions: (a) under moderate block lengths, TurboAE approaches state-of-the-art performance under canonical channels; (b) moreover, TurboAE outperforms the state-of-the-art codes under non-canonical settings in terms of reliability. TurboAE shows that the development of channel coding design can be automated via deep learning, with near-optimal performance.

This paper presents a Offset Aperture (OA) based single camera system and proposes a optimized vision processor, a new hardware architecture for fast, low-energy, and low-complexity depth extraction. The proposed design was fabricated in 110nm CMOS image sensor technology and supports 32-level depth resolution on 1920×1080 full HD image with 30fps, consuming 280.53mW from 1.5V supply and a mere 2.8% of bad classification. The low-complexity algorithms are employed to eliminate the DRAM access, thereby the proposed OA architecture can be directly embedded with the CMOS image sensor and commercial image processing chip.

The design of codes for communicating reliably over a statistically well defined channel is an important endeavor involving deep mathematical research and wide- ranging practical applications. In this work, we present the first family of codes obtained via deep learning, which significantly beats state-of-the-art codes designed over several decades of research. The communication channel under consideration is the Gaussian noise channel with feedback, whose study was initiated by Shannon; feedback is known theoretically to improve reliability of communication, but no practical codes that do so have ever been successfully constructed. We break this logjam by integrating information theoretic insights harmoniously with recurrent-neural-network based encoders and decoders to create novel codes that outperform known codes by 3 orders of magnitude in reliability. We also demonstrate several desirable properties in the codes: (a) generalization to larger block lengths; (b) composability with known codes; (c) adaptation to practical constraints. This result also presents broader ramifications to coding theory: even when the channel has a clear mathematical model, deep learning methodologies, when combined with channel specific information-theoretic insights, can potentially beat state-of-the-art codes, constructed over decades of mathematical research.

Discovering a correlation from one variable to another variable is of fundamental scientific and practical interest. While existing correlation measures are suitable for discovering average correlation, they fail to discover hidden or potential correlations. To bridge this gap, (i) we postulate a set of natural axioms that we expect a measure of potential correlation to satisfy; (ii) we show that the rate of information bottleneck, i.e., the hypercontractivity coefficient, satisfies all the proposed axioms; (iii) we provide a novel estimator to estimate the hypercontractivity coefficient from samples; and (iv) we provide numerical experiments demonstrating that this proposed estimator discovers potential correlations among various indicators of WHO datasets, is robust in discovering gene interactions from gene expression time series data, and is statistically more powerful than the estimators for other correlation measures in binary hypothesis testing of canonical examples of potential correlations.

This paper shows that the slope at each corner point of the capacity region of the general broadcast channel coincides with that of the randomized time division (hence the Marton) inner bound and the Nair-El Gamal (as well as the Körner-Marton) outer bound. We then show that the optimal superposition coding inner bound by Bandemer, El Gamal, and Kim can be simplified to the convex closure of the union of the Cover-Bergmans UX region and the Cover-van der Meulen UV region. Generalizing a result by Hajek and Pursely on the skewed binary symmetric broadcast channel, we show that for binary input broadcast channels, the UV region reduces to time division further simplifying the superposition coding inner bound. Finally we establish necessary and sufficient conditions for the optimality of the superposition inner bound for skewed binary broadcast channels.

We establish the capacity region of several classes of broadcast channels with random state in which the channel to each user is selected from two possible channel state components and the state is known only at the receivers. When the channel components are deterministic, we show that the capacity region is achieved via Marton coding. This channel model does not belong to any class of broadcast channels for which the capacity region was previously known and is useful in studying wireless communication channels when the fading state is known only at the receivers. We then establish the capacity region when the channel components are ordered, e.g., degraded. In particular, we show that the capacity region for the broadcast channel with degraded Gaussian vector channel components is attained via Gaussian input distribution. Finally, we extend the results on ordered channels to two broadcast channel examples with more than two channel components, but show that these extensions do not hold in general.

This paper shows that the capacity region of the continuous-time Poisson broadcast channel is achieved via superposition coding for most channel parameter values. Interestingly, the channel in some subset of these parameter values does not belong to any of the existing classes of broadcast channels for which superposition coding is optimal (e.g., degraded, less noisy, and more capable). In particular, we introduce the notion of effectively less noisy broadcast channel and show that it implies less noisy but is not in general implied by more capable. For the rest of the channel parameter values, we show that there is a gap between Marton’s inner bound and the UV outer bound.

This paper shows that the capacity region of the continuous-time Poisson broadcast channel is achieved via superposition coding for most channel parameter values. Interestingly, the channel in some subset of these parameter values does not belong to any of the existing classes of broadcast channels for which superposition coding is optimal (e.g., degraded, less noisy, more capable). For the rest of the channel parameter values, we show that there is a gap between Marton's inner bound and the UV outer bound.

This paper shows that the Maddah-Ali–Tse (MAT) scheme, which achieves the symmetric capacity of two example broadcast channels with strictly causal state information at the transmitter, is a simple special case of the Shayevitz–Wigger (SW) scheme for the broadcast channel with generalized feedback, which involves block Markov coding, Gray–Wyner compression, superposition coding, and Marton coding. Focusing on the class of symmetric broadcast channels with state, we derive an expression for the maximum achievable symmetric rate using the SW scheme. We show that the MAT results for the two-receiver case can be recovered by evaluating this expression for the special case in which superposition coding and Marton coding are not used. We then introduce a new broadcast channel example that shares many features of the MAT examples. We show that another special case of our maximum symmetric rate expression in which superposition coding is also used attains a higher symmetric rate than the MAT scheme. The symmetric capacity of this new example is not known, however.

This paper establishes the capacity region of a class of broadcast channels with random state in which each channel component is selected from two possible functions and each receiver knows its state sequence. This channel model does not fit into any class of broadcast channels for which the capacity region was previously known and is useful in studying wireless communication channels when the fading state is known only at the receivers. The capacity region is shown to coincide with the UV outer bound and is achieved via Marton coding.