It is our pleasure to share with you this Special Issue, which brings together a diverse set of articles dealing with various aspects of semantic and goal-oriented communications, providing a snapshot of research activities in this highly active research area. Wireless communications and networking research has traditionally focused on improving the capacity and throughput of the underlying wireless network. However, recent explosion in data-driven machine learning applications and their reliance on huge datasets collected by edge devices have raised legitimate concerns that the increasing data traffic might soon overwhelm the capacity of current networks despite ongoing efforts to increase their capacity and efficiency. Also, most of the edge intelligence applications impose stringent delay constraints, which cannot be met by naive forwarding of data samples for processing at the receiver end. This made it obvious to researchers in both academia and industry that it is essential to analyze the “value” or “relevance” of collected data, and filter and prioritize the delivery of data based on its value/relevance as well as the wireless channel and network conditions. In this context, data value will be closely connected to the underlying signals and processes that generate the data, e.g., text, image, video, or sensor data, and what the receiver intends to do with the received data. This subjectivity of data value makes semantic and goal-oriented communication a rather elusive research topic, which has led to both an increasingly rich and active area of investigation, but also a controversial one, mainly due to the lack of clear and widely agreed-upon definitions of some of the core concepts and formulations. Despite these disagreements, there is almost unanimous consensus on the importance and potential impact of this line of investigation for the design of future communication systems and networks.

Communication systems to date primarily aim at reliably communicating bit sequences. Such an approach provides efficient engineering designs that are agnostic to the meanings of the messages or to the goal that the message exchange aims to achieve. Next generation systems, however, can be potentially enriched by folding message semantics and goals of communication into their design. Further, these systems can be made cognizant of the context in which communication exchange takes place, thereby providing avenues for novel design insights. This tutorial summarizes the efforts to date, starting from its early adaptations, semantic-aware and task-oriented communications, covering the foundations, algorithms and potential implementations. The focus is on approaches that utilize information theory to provide the foundations, as well as the significant role of learning in semantics and task-aware communications.

The problem of distributed representation learning is one in which multiple sources of information X1,..., XKare processed separately so as to learn as much information as possible about some ground truth Y. We investigate this problem from information-theoretic grounds, through a generalization of Tishby's centralized Information Bottleneck (IB) method to the distributed setting. Specifically, K encoders, K ≥ 2, compress their observations X1,..., XKseparately in a manner such that, collectively, the produced representations preserve as much information as possible about Y. We study both discrete memoryless (DM) and memoryless vector Gaussian data models. For the discrete model, we establish a single-letter characterization of the optimal tradeoff between complexity (or rate) and relevance (or information) for a class of memoryless sources (the observations X1,..., XKbeing conditionally independent given Y). For the vector Gaussian model, we provide an explicit characterization of the optimal complexity-relevance tradeoff. Furthermore, we develop a variational bound on the complexity-relevance tradeoff which generalizes the evidence lower bound (ELBO) to the distributed setting. We also provide two algorithms that allow to compute this bound: i) a Blahut-Arimoto type iterative algorithm which enables to compute optimal complexity-relevance encoding mappings by iterating over a set of self-consistent equations, and ii) a variational inference type algorithm in which the encoding mappings are parametrized by neural networks and the bound approximated by Markov sampling and optimized with stochastic gradient descent. Numerical results on synthetic and real datasets are provided to support the efficiency of the approaches and algorithms developed in this paper.

In this paper, we study the vector Gaussian Chief Executive Officer (CEO) problem under logarithmic loss distortion measure. Specifically, K ≥ 2 agents observe independently corrupted Gaussian noisy versions of a remote vector Gaussian source, and communicate independently with a decoder or CEO over rate-constrained noise-free links. The CEO also has its own Gaussian noisy observation of the source and wants to reconstruct the remote source to within some prescribed distortion level where the incurred distortion is measured under the logarithmic loss penalty criterion. We find an explicit characterization of the rate-distortion region of this model. The result can be seen as the counterpart to the vector Gaussian setting of that by Courtade-Weissman which provides the rate-distortion region of the model in the discrete memoryless setting. For the proof of this result, we obtain an outer bound by means of a technique that relies on the de Bruijn identity and the properties of Fisher information. The approach is similar to Ekrem-Ulukus outer bounding technique for the vector Gaussian CEO problem under quadratic distortion measure, for which it was there found generally non-tight; but it is shown here to yield a complete characterization of the region for the case of logarithmic loss measure. Also, we show that Gaussian test channels with time-sharing exhaust the Berger-Tung inner bound, which is optimal. Furthermore, application of our results allows us to find the complete solutions of two related problems: a quadratic vector Gaussian CEO problem with determinant constraint and the vector Gaussian distributed Information Bottleneck problem. Finally, we develop Blahut-Arimoto type algorithms that allow to compute numerically the regions provided in this paper, for both discrete and Gaussian models. With the known relevance of the logarithmic loss fidelity measure in the context of learning and prediction, the proposed algorithms may find usefulness in a variety of applications where learning is performed distributively. We illustrate the efficiency of our algorithms through some numerical examples.

We study a class of distributed hypothesis testing against conditional independence problems. Under the criterion that stipulates minimization of the Type II error exponent subject to a (constant) upper bound c on the Type I error rate, we characterize the set of encoding rates and exponent for both discrete memoryless and memoryless vector Gaussian settings.

In this paper, we study the vector Gaussian Chief Executive Officer (CEO) problem under logarithmic loss distortion measure. Specifically, K > 2 agents observe independently corrupted Gaussian noisy versions of a remote vector Gaussian source, and communicate independently with a decoder or CEO over rate-constrained noise-free links. The CEO wants to reconstruct the remote source to within some prescribed distortion level where the incurred distortion is measured under the logarithmic loss penalty criterion. We find an explicit characterization of the rate-distortion region of this model. For the proof of this result, we obtain an outer bound on the region of the vector Gaussian CEO problem by means of a technique that relies on the de Bruijn identity and the properties of Fisher information. The approach is similar to Ekrem-Ulukus outer bounding technique for the vector Gaussian CEO problem under quadratic distortion measure, for which it was there found generally non-tight; but it is shown here to yield a complete characterization of the region for the case of logarithmic loss measure. Also, we show that Gaussian test channels with time-sharing exhaust the BergerTung inner bound, which is optimal. Furthermore, we also show that the established result under logarithmic loss provides an outer bound for a quadratic vector Gaussian CEO problem with determinant constraint, for which we characterize the optimal rate-distortion region.

We study the transmission over a network in which users send information to a remote destination through relay nodes that are connected to the destination via finite-capacity error-free links, i.e., a cloud radio access network. The relays are constrained to operate without knowledge of the users' codebooks, i.e., they perform oblivious processing. The destination, or central processor, however, is informed about the users' codebooks. We establish a single-letter characterization of the capacity region of this model for a class of discrete memoryless channels in which the outputs at the relay nodes are independent given the users' inputs. We show that both relaying à-la Cover-El Gamal, i.e., compress-and-forward with joint decompression and decoding, and “noisy network coding” are optimal. The proof of the converse part establishes, and utilizes, connections with the Chief Executive Officer source coding problem under logarithmic loss distortion measure. Extensions to general discrete memoryless channels are also investigated. In this case, we establish the inner and outer bounds on the capacity region. For memoryless Gaussian channels within the studied class of channels, we characterize the capacity region when the users are constrained to time-share among Gaussian codebooks. Furthermore, we also discuss the suboptimality of separate decompression and decoding and the role of time sharing.

In this paper, we present iterative algorithms that numerically compute the rate-distortion regions of two problems: the two-encoder multiterminal source coding problem and the Chief Executive Officer (CEO) problem, both under logarithmic loss distortion measure. With the clear connection of these models with the distributed information bottleneck method, the proposed algorithms may find usefulness in a variety of applications, such as clustering, pattern recognition and learning. We illustrate the efficiency of our algorithms through some numerical examples.

We study the problem of receive beamforming in uplink cascade multiple-input multiple-output (MIMO) systems as an instance of that of cascade multiterminal source coding for lossy function computation. Using this connection, we develop two coding schemes for the second and show that their application leads to beamforming schemes for the first. In the first coding scheme, each terminal in the cascade sends a description of the source that it observes; the decoder reconstructs all sources, lossily, and then computes an estimate of the desired function. This scheme improves upon standard routing in that every terminal only compresses the innovation of its source w.r.t. the descriptions that are sent by the previous terminals in the cascade. In the second scheme, the desired function is computed gradually in the cascade network, and each terminal sends a finer description of it. In the context of uplink cascade MIMO systems, the application of these two schemes leads to centralized receive-beamforming and distributed receive-beamforming, respectively. Numerical results illustrate the performance of the proposed methods and show that they outperform standard routing.

We study the transmission over a cloud radio access network in which multiple base stations (BS) are connected to a central processor (CP) via finite-capacity backhaul links. We propose a lattice-based coding scheme in which the BSs decode linear combinations of the transmitted messages, in the spirit of Compute-and-Forward (CoF), but differs from it essentially in that the decoded equations are remapped to linear combinations of the channel input symbols, sent compressed in a lossy manner to the CP, and are not required to be linearly independent. Also, by opposition to the standard CoF, an appropriate multi-user decoder is utilized to recover the sent messages. This novel scheme differs from both classical Compute-and-Forward and Successive Wyner-Ziv (SWZ) and it is shown to outperform both schemes, in certain regimes, through some numerical examples.

We study the transmission over a cloud radio access network in which multiple relays are connected to a central processor (CP) via error-free finite-capacity links. We develop a lattice-based coding scheme in which each relay node computes a linear combination of the users' messages, in the spirit of standard compute-and-forward (CoF). However, rather than forwarding the computed equation to the CP as in standard CoF, the relay first maps this equation into one on the users' input symbols and then compresses it jointly with its channel output. The equations need not be linearly independent, and the compression also takes into account the correlation with the equations and channel outputs at other relay nodes through Wyner-Ziv coding. The CP first decompresses the signals and then decodes the users' messages successively. We analyze the sum-rate offered by this coding scheme, and show that it outperforms successive Wyner-Ziv scheme in all regimes, and it improves strictly upon the best of successive Wyner-Ziv and CoF in certain regimes.

We study the transmission over a cloud radio access network in which multiple base stations, acting as relay nodes, are connected to a central processor (CP) via error-free rate-limited backhaul links. We propose two lattice-based coding schemes. In the first scheme, each relay node decodes linear combinations of the users' messages in the spirit of compute-and-forward (CoF), but departs from it essentially in that the decoded equations are remapped to equations on the users' input symbols, sent compressed in a lossy manner to the CP, and are not required to be linearly independent. The compression accounts for the correlation between equations at the relay nodes through Wyner-Ziv coding. Also, by opposition to the standard CoF, an appropriate multi-user decoder is utilized to recover the sent messages. The second scheme generalizes the first one by also allowing, at each relay node, a joint compression of the decoded equation on the users' input symbols and the received signal. Both schemes apply in general, but are especially suited for situations in which there are more users that relays. We show that both schemes can outperform standard CoF and successive Wyner-Ziv schemes in certain regimes, and illustrate the gains through some numerical examples.

Transmission of a Gaussian source over a time-varying Gaussian channel is studied in the presence of time-varying correlated side information at the receiver. A block fading model is considered for both the channel and the side information, whose states are assumed to be known only at the receiver. The optimality of separate source and channel coding in terms of average end-to-end distortion is shown when the channel is static, while the side information state follows a discrete or a continuous and quasiconcave distribution. When both the channel and side information states are time-varying, separate source and channel coding is suboptimal in general. A partially informed encoder lower bound is studied by providing the channel state information to the encoder. Several achievable transmission schemes are proposed based on uncoded transmission, separate source and channel coding, joint decoding, and hybrid digital-analog transmission. Uncoded transmission is shown to be optimal for a class of continuous and quasiconcave side information state distributions, while the channel gain may have an arbitrary distribution. To the best of our knowledge, this is the first example, in which the uncoded transmission achieves the optimal performance thanks to the time-varying nature of the states, while it is suboptimal in the static version of the same problem. Then, the optimal distortion exponent, which quantifies the exponential decay rate of the expected distortion in the high SNR regime, is characterized for Nakagami distributed channel and side information states, and it is shown to be achieved by hybrid digital-analog and joint decoding schemes in certain cases, illustrating the suboptimality of pure digital or analog transmission in general.

The class of orthogonal relay channels in which the orthogonal channels connecting the source terminal to the relay and the destination, and the relay to the destination, depend on a state sequence, is considered. It is assumed that the state sequence is fully known at the destination, while it is not known at the source or the relay. The capacity of this class of relay channels is characterized, and shown to be achieved by the partial decode-compress-and-forward (pDCF) scheme. Then, the capacity of certain binary and Gaussian state-dependent orthogonal relay channels are studied in detail, and it is shown that the compress-and-forward (CF) and partial-decode-and-forward (pDF) schemes are suboptimal in general. To the best of our knowledge, this is the first single relay channel model for which the capacity is achieved by pDCF, while pDF and CF schemes are both suboptimal. Furthermore, it is shown that the capacity of the considered class of state-dependent orthogonal relay channels is in general below the cut-set bound. The conditions under which pDF or CF suffices to meet the cut-set bound, and hence, achieve the capacity, are also derived.

Transmission of a Gaussian source over a time-varying multiple-input multiple-output (MIMO) channel is studied under strict delay constraints. Availability of a correlated side information at the receiver is assumed, whose quality, i.e., its correlation with the source signal, also varies over time. A block-fading model is considered for the states of the time-varying channel and side information; perfect state information at the receiver is assumed, while the transmitter knows only the statistics. The high signal to noise ratio performance, characterized by the distortion exponent, is studied for this joint source-channel coding problem. An upper bound is derived and compared with several lower bounds based on list decoding (LD), hybrid digital-analog transmission, as well as multi-layer schemes, which transmit successive refinements of the source, relying on progressive or superposition transmission with LD. The optimal distortion exponent is characterized for the single-input multiple-output and multiple-input single-output scenarios by showing that the distortion exponent achieved by multi-layer superposition encoding with joint decoding meets the proposed upper bound. In the MIMO scenario, the optimal distortion exponent is characterized in the low bandwidth ratio regime, and it is shown that the multi-layer superposition encoding performs very close to the upper bound in the high bandwidth ratio regime.

Transmission of correlated Gaussian sources over a Gaussian interference channel is studied. Each terminal has one source available, which has to be reconstructed at the corresponding destination with the minimum average distortion. Focusing on the strong interference setting, we first derive necessary conditions on the achievable distortion pairs. Then, focusing on the symmetric scenario, we present achievable distortion pairs considering several transmission strategies. We compare the achievable distortion by the proposed schemes with the lower bound. In particular, we consider separate source and channel coding, uncoded transmission, a vector quantization (VQ) scheme which uses the quantization codewords as channel inputs, and finally a superposition of two quantization codewords. We show that the VQ scheme is optimal in the high SNR regime. We also show that the proposed superposition scheme performs very close to the lower bound in certain regimes.

Systematic lossy transmission of a Gaussian source over a time-varying Gaussian channel is considered. A noisy version of the source is transmitted in an uncoded fashion to the destination through a time-varying Gaussian “base channel”, constituting the systematic part of the transmission. A second time-varying Gaussian channel, called the “enhancement channel”, orthogonal to the first one, is available for coded transmission of the source sequence. A block fading model for both channels is considered, and the average end-to-end distortion is studied assuming perfect channel state information only at the receiver. It is shown that if the enhancement channel is static and the base channel gain has a discrete or continuous-quasiconcave distribution, then the separation theorem applies. However, when both channels are block fading, separation theorem does not hold anymore. A lower bound is obtained by providing the enhancement channel state to the encoder, and it is shown that uncoded transmission is exactly optimal for certain base channel fading distributions, while the enhancement channel fading has arbitrary distribution. A joint decoding scheme is also presented and is shown to outperform uncoded transmission and separate source and channel coding for other base channel distributions.

The class of orthogonal relay channels with state in which the source and the relay are connected through a channel that depends on a state sequence is considered. It is assumed that the state sequence is fully known at the destination while it is not known at the source or the relay. The source and the relay are connected to the destination through orthogonal channels. The capacity of this class of relay channels is characterized and it is shown to be achieved by the partial decode-compress-and-forward (pDCF) scheme. To the best of our knowledge, this is the first single relay channel model for which the capacity is achieved by pDCF, while partial decode-and-forward (pDF) and compress-and-forward (CF) schemes are suboptimal in general.

We consider the joint source-channel coding problem of sending a Gaussian source over a single input-single output (SISO) fading channel when the decoder has additional correlated side information whose quality is also time-varying. We assume a block fading model for both the channel and side information qualities, and assume perfect state information at the receiver, while the transmitter has only a statistical knowledge. We are interested in the expected squared-error distortion for this system. We study separate source-channel coding,uncoded transmission and a joint source-channel transmission scheme based on joint decoding at the receiver. We then extend joint decoding scheme technique to hybrid digital-analog and multi-layer schemes. We provide numerical results in the finite SNR regime, and derive closed form expressions for the distortion exponent in the high SNR regime.

The joint source-channel coding problem of sending a Gaussian source over a multiple input-multiple output (MIMO) fading channel with time-varying correlated side information at the decoder is studied. A block fading model for both the channel and the side information qualities is considered, and perfect (channel and side information) state information at the receiver is assumed, while the transmitter has only a statistical knowledge. In particular, the high SNR performance is studied by deriving the distortion exponent of various transmission schemes. An upper bound on the distortion exponent is derived by providing the channel state to the encoder while the side information state remains unknown. Separate source and channel coding is considered as well as joint decoding at the receiver. The joint decoding scheme is extended to multiple digital layers. Finally, a hybrid digital-analog (HDA) scheme is analyzed. While the optimal distortion exponent is completely characterized for MISO/SIMO channels, for general MIMO channels, the optimal distortion exponent is characterized in the small bandwidth ratio regime.

For the problem of source transmission with fading channel and side information quality, we have characterized the optimal distortion exponent for SISO systems by deriving an upper bound and providing transmission schemes that meet this bound. We also have provided closed-form expressions for the distortion exponents of the presented schemes. Interestingly, for x <; b <; 1, the optimal distortion exponent coincides with the optimal distortion exponent when side information is absent.

The one-helper joint source-channel coding problem, in which a main source is to be reconstructed with minimum distortion with the help of a correlated helper source, is studied. Focusing on the case of Gaussian sources and a Gaussian multiple access channel (MAC), a generalized hybrid digital-analog scheme is proposed, in which each user allocates its available power among the analog and digital signals and transmits a superposition of the two. It is shown that this generalized hybrid scheme reduces to pure analog or pure digital transmission depending on the system parameters. Finally, the optimal hybrid transmission strategy is identified analytically in certain special scenarios modeling legacy systems.