In lossy compression, Wang et al. [1] recently introduced the rate-distortion-perception-classification function, which supports multi-task learning by jointly optimizing perceptual quality, classification accuracy, and reconstruction fidelity. Building on the concept of a universal encoder introduced in [2], we investigate the universal representations that enable a broad range of distortion-classification tradeoffs through a single shared encoder coupled with multiple task-specific decoders. We establish, through both theoretical analysis and numerical experiment, that for a Gaussian source under mean-squared error (MSE) distortion, the distortion-classification tradeoff region can be achieved using a single universal encoder. For general sources, we characterize the achievable region and identify conditions under which a universal encoder can produce a small distortion penalty. The experimental result on the MNIST dataset further supports our theoretical findings. We show that universal encoders can obtain distortion performance comparable to task-specific encoders. These results demonstrate the practicality and effectiveness of the proposed universal framework in multi-task compression scenarios.

Given the training data and labels from several seen domains, Domain Generalization (DG) aims to learn models that generalize well on unlabeled data from unseen domains. Due to the distribution of data and/or labels may vary between domains, current DG methods are mainly based on the theme of domain-invariant representation learning which decomposes the learning process into two steps: (1) finding a representation function from the input space to the representation/feature space to learn the so-called domain-invariant features, i.e., the features that are statistically stable between domains, and (2) designing a classifier on top of these domain-invariant features which is mutually optimal for all domains. Recent works on DG show that Contrastive Learning (CL) which was originally proposed to learn a representation function such that similar samples are pulled closer while dissimilar samples are pushed far away in the representation/feature space, appears as a promising solution for domain-invariant features learning. In this paper, we first establish the fundamental theory to justify why CL might be a potential approach for DG by showing that CL, under particular settings, allows a mechanism to minimize a loss function that enforces an optimal classifier for all training domains. Based on these foundations, we revisit the recent works that apply CL for DG to indicate their limitations and suggest a modification by combining the CL method with current DG methods for better out-of-domain generalization. Our numerical results point out that our proposed algorithm can achieve state-of-the-art performance on multiple DG benchmark datasets.

Through minimization of an appropriate loss function such as the InfoNCE loss, contrastive learning (CL) learns a useful representation function by pulling positive samples close to each other while pushing negative samples far apart in the embedding space. The positive samples are typically created using "label-preserving" augmentations, i.e., domain-specific transformations of a given datum or anchor. In absence of class information, in unsupervised CL (UCL), the negative samples are typically chosen randomly and independently of the anchor from a preset negative sampling distribution over the entire dataset. This leads to class-collisions in UCL. Supervised CL (SCL), avoids this class collision by conditioning the negative sampling distribution to samples having labels different from that of the anchor. In hard-UCL (H-UCL), which has been shown to be an effective method to further enhance UCL, the negative sampling distribution is conditionally tilted, by means of a hardening function, towards samples that are closer to the anchor. Motivated by this, in this paper we propose hard-SCL (H-SCL) wherein the class conditional negative sampling distribution is tilted via a hardening function. Our simulation results confirm the utility of H-SCL over SCL with significant performance gains in downstream classification tasks. Analytically, we show that in the limit of infinite negative samples per anchor and a suitable assumption, the H-SCL loss is upper bounded by the H-UCL loss, thereby justifying the utility of H-UCL for controlling the H-SCL loss in the absence of label information. Through experiments on several datasets, we verify the assumption as well as the claimed inequality between H-UCL and H-SCL losses. We also provide a plausible scenario where H-SCL loss is lower bounded by UCL loss, indicating the limited utility of UCL in controlling the H-SCL loss.1

This paper investigates a beamforming scheme designed to minimize the symbol error probability (SEP) for a legitimate user while guaranteeing that the likelihood of an eavesdropper correctly recovering symbols remains below a predefined threshold. The focus is on finding an optimal beamforming vector for binary antipodal signal detection in multiple-input multiple-output (MIMO) Gaussian wiretap channels. Finding the optimal beamforming vector is a non-convex problem, and thus conventional computationally efficient algorithms for convex problems cannot be applied in this context. To that end, our proposed algorithm relies on Karush–Kuhn–Tucker (KKT) conditions and the generalized eigen-decomposition method to find an exact solution. The numerical results are presented to assess the performance of the proposed method for various scenarios.

Domain generalization aims to learn a model with good generalization ability, that is, the learned model should not only perform well on several seen domains but also on unseen domains with different data distributions. State-of-the-art domain generalization methods typically train a representation function followed by a classifier jointly to minimize both the classification risk and the domain discrepancy. However, when it comes to model selection, most of these methods rely on traditional validation routines that select models solely based on the lowest classification risk on the validation set. In this paper, we theoretically demonstrate a trade-off between minimizing classification risk and mitigating domain discrepancy, i.e., it is impossible to achieve the minimum of these two objectives simultaneously. Motivated by this theoretical result, we propose a novel model selection method suggesting that the validation process should account for both the classification risk and the domain discrepancy. We validate the effectiveness of the proposed method by numerical results on several domain generalization datasets.

We consider a communication channel whose input is modeled as a discrete random variable X with distribution pX. X is transmitted over a noisy channel and distorted by a continuous-valued noise to result in a continuous-valued output signal U at the receiver. A thresholding quantizer Q is applied to reconstruct a discrete signal V = Q(U) from the continuous-valued U. Our goal is to jointly design both the input distribution pXand the thresholding quantizer Q to maximize the mutual information I(X; V) between the input X and V since the accuracy of any decoding algorithm that estimates X from V fundamentally depends on I(X; V). In this paper, an alternating maximization algorithm is proposed that guarantees to achieve a locally optimal solution. In addition, we numerically show that by randomly selecting a set of initial starting points, the proposed algorithm is capable of achieving the globally optimal solution. Both the theoretical and numerical results are provided to justify our approach.

Assessing the cognitive workload of human interactants in mixed-initiative teams is a critical capability for autonomous interactive systems to enable adaptations that improve team performance. Yet, it is still unclear, due to diverging evidence, which sensing modality might work best for the determination of human workload. In this paper, we report results from an empirical study that was designed to answer this question by collecting eye gaze and electroencephalogram (EEG) data from human subjects performing an interactive multi-modal driving task. Different levels of cognitive workload were generated by introducing secondary tasks like dialogue, braking events, and tactile stimulation in the course of driving. Our results show that pupil diameter is a more reliable indicator for workload prediction than EEG. And more importantly, none of the five different machine learning models combining the extracted EEG and pupil diameter features were able to show any improvement in workload classification over eye gaze alone, suggesting that eye gaze is a sufficient modality for assessing human cognitive workload in interactive, multi-modal, multi-task settings.

Domain generalization (DG) is a branch of transfer learning that aims to train the learning models on several seen domains and subsequently apply these pre-trained models to other unseen (unknown but related) domains. To deal with challenging settings in DG where both data and label of the unseen domain are not available at training time, the most common approach is to design the classifiers based on the domain-invariant representation features, i.e., the latent representations that are unchanged and transferable between domains. Contrary to popular belief, we show that designing classifiers based on invariant representation features alone is necessary but insufficient in DG. Our analysis indicates the necessity of imposing a constraint on the reconstruction loss induced by representation functions to preserve most of the relevant information about the label in the latent space. More importantly, we point out the trade-off between minimizing the reconstruction loss and achieving domain alignment in DG. Our theoretical results motivate a new DG framework that jointly optimizes the reconstruction loss and the domain discrepancy. Both theoretical and numerical results are provided to justify our approach.

Invariance-principle-based methods such as Invariant Risk Minimization (IRM), have recently emerged as promising approaches for Domain Generalization (DG). Despite promising theory, such approaches fail in common classification tasks due to mixing of true invariant features and spurious invariant features1. To address this, we propose a framework based on the conditional entropy minimization (CEM) principle to filter-out the spurious invariant features leading to a new algorithm with a better generalization capability. We show that our proposed approach is closely related to the well-known Information Bottleneck (IB) framework and prove that under certain assumptions, entropy minimization can exactly recover the true invariant features. Our approach provides competitive classification accuracy compared to recent theoretically-principled state-of-the-art alternatives across several DG datasets.

Partitioning algorithms play a key role in many scientific and engineering disciplines. A partitioning algorithm divides a set into a number of disjoint subsets or partitions. Often, the quality of the resulted partitions is measured by the amount of impurity in each partition, the smaller impurity the higher quality of the partitions. Let M be the number of N-dimensional elements in a set and K be the number of desired partitions, then an exhaustive search over all the possible partitions to find a minimum partition has the complexity of O(KM) which quickly becomes impractical for many applications with modest values of K and M. Thus, many approximate algorithms with polynomial time complexity have been proposed, but few provide the bounded guarantee. In this paper, we propose a linear time algorithm with bounded guarantee based on the maximum likelihood principle. Furthermore, the guarantee bound of the proposed algorithm is better than the state-of-the-art method in [1] for many impurity functions, and at the same time, for K ≥ N, the computational complexity is reduced from O(M3) to O(M).

Set partitioning is a key component of many algorithms in machine learning, signal processing and communications. In general, the problem of finding a partition that minimizes a given impurity (loss function) is NP-hard. As such, there exists a wealth of literature on approximate algorithms and theoretical analysis for the partitioning problem under different settings. In this paper, we formulate and solve a variant of the partition problem called the minimum weighted concave impurity partition under constraint (MIPUC). MIPUC finds an optimal partition that minimizes a given weighted concave loss function under a given concave constraint. MIPUC generalizes the recently proposed Deterministic Information Bottleneck problem which finds an optimal partition that maximizes the mutual information between the input and partitioned output while minimizing the partitioned output entropy. Our proposed algorithm is based on an optimality condition, which allows us to find a locally optimal solution efficiently. We also show that the optimal partitions are separated by some hyperplanes in the space of posterior probability mass functions.

In this paper, we consider the problem of one-bit (two-level) output-quantization maximizing mutual information between quantizer-output and channel-input using a single threshold for a discrete signal that is corrupted by a continuous additive noise. A necessary condition is constructed for which the thresholding quantizer is optimal. In addition, we show that if the distribution of the additive noise satisfies a mild condition, then a global optimal threshold can be found efficiently via a modified fixed-point algorithm.

Consider a channel whose the input alphabet set$\mathbb {X}=\{x_{1},x_{2}, {\dots },x_{K}\}$contains$K$discrete symbols modeled as a discrete random variable$X$having a probability mass function$\mathbf {p}(\mathbf {x}) = [p(x_{1}), p(x_{2}), {\dots }, p(x_{K})]$and the received signal$Y$being a continuous random variable.$Y$is a distorted version of$X$caused by a channel distortion, characterized by the conditional densities$p(y|x_{i})=\phi _{i}(y)$,$i=1,2, {\dots },K$. To recover$X$, a quantizer$Q$is used to quantize$Y$back to a discrete output$\mathbb {Z} =\{z_{1}, z_{2}, {\dots }, z_{N}\}$corresponding to a random variable$Z$with a probability mass function$\mathbf {p}(\mathbf {z}) = [p(z_{1}), p(z_{2}), {\dots }, p(z_{N})]$such that the mutual information$I(X;Z)$is maximized subject to an arbitrary constraint on$\mathbf {p}(\mathbf {z})$. Formally, we are interested in designing an optimal quantizer$Q^{*}$that maximizes$\beta I(X;Z) - C(Z)$where$\beta $is a positive number that controls the trade-off between maximizing$I(X;Z)$and minimizing an arbitrary cost function$C(Z)$. Let$\mathbf {p}(\mathbf {x}|y)=[p(x_{1}|y),p(x_{2}|y), {\dots },p(x_{K}|y)]$be the posterior distribution of$X$for a given value of$y$, we show that for any arbitrary cost function$C(.)$, the optimal quantizer$Q^{*}$separates the vectors$\mathbf {p}(\mathbf {x}|y)$into convex regions. Using this result, a method is proposed to determine an upper bound on the number of thresholds (decision variables on$y$) which is used to speed up the algorithm for finding an optimal quantizer. Numerical results are presented to validate the findings.

Given an original discrete source X with the distribution pX that is corrupted by noise to produce a noisy data Y with the given joint distribution p(X,Y). A quantizer/classifier Q : Y → Z is then used to classify/quantize Y to a discrete partitioned output Z having probability distribution pZ. Next, Z is transmitted over a discrete memoryless channel (DMC) with a given channel matrix A that produces the final discrete output T . One wants to design an optimal quantizer/classifier Q* to minimize the end-to-end impurity/cost function F(X, T) between the input X and the final output T. Our result generalizes some previous results. First, an iteration linear time complexity algorithm is proposed to find the locally optimal quantizer. Second, we show that the optimal quantizers produce the hard partitions that are equivalent to the cuts by hyper-planes in the space of the posterior distribution pX|Y. This result provides a polynomial-time complexity algorithm to find the globally optimal quantizer. Finally, in the special case where the source X is binary, an efficient algorithm is proposed to find the truly global optimal partition.

Partitioning algorithms play a key role in machine learning, signal processing, and communications. They are used in many well-known NP-hard problems such as k-means clustering and vector quantization. The goodness of a partition scheme is measured by a given impurity function over the resulted partitions. The optimal partition is one(s) with the minimum impurity. Practical algorithms for finding an optimal partitioning are approximate, heuristic, and often assume certain properties of the given impurity function such as concavity/convexity. In this paper, we propose a heuristic, efficient (linear time) algorithm for finding the minimum impurity for a broader class of impurity functions which includes popular impurities such as Gini index and entropy. We also make a connection to a well-known result which states that the optimal partitions correspond to the regions separated by hyperplane cuts in the probability space of the posterior distribution.

In this paper, we consider a channel whose the input is a binary random source X ∈ {x1,x2} with the probability mass function (pmf) pX= [px1,px2] and the output is a continuous random variable Y ∈ R as a result of a continuous noise, characterized by the channel conditional densities py|x1= φ1(y) and py|x2= φ2(y). A quantizer Q is used to map Y back to a discrete set Z ∈ {z1,z2,...,zN}. To retain most amount of information about X, an optimal Q is one that maximizes I(X;Z). On the other hand, our goal is not only to recover X but also ensure that pZ= [pz(1),pz(2),...,pz(N)] satisfies a certain constraint. In particular, we are interested in designing a quantizer that maximizes βI(X;Z)-C(pZ) where β is a tradeoff parameter and C(pZ) is an arbitrary cost function of pZ. Let the posterior probability px(1)|y= ry= [px(1)φ1(y)]/([px(1)φ1(y)] + [px(2)φ2(y)]), our result shows that the structure of the optimal quantizer separates ryinto convex cells. In other words, the optimal quantizer has the form: Q*(ry) = zi, if a8i-1≤ ryi* for some optimal thresholds a0* = 01*2*N-1*N* = 1. Based on this optimal structure, we describe some fast algorithms for determining the optimal quantizers.

We consider a channel with discrete input X, a continuous noise that corrupts the input X to produce the continuous-valued output U. A thresholding quantizer is then used to quantize the continuous-valued output U to the final discrete output V. The goal is to jointly design a thresholding quantizer that maximizes the mutual information between input and quantized output I(X; V ) while minimizing a pre-specified function of the quantized output F(pV). A general dynamic programming algorithm is proposed having the time complexity O(KNM2) where N, M and K are the sizes of input X, output U and quantized output V, respectively. Moreover, we show that if F(pV) Σi=1Kgi(p(vi)) where gi(.) is a convex function, p(vi) ∈ pV{pv1,..., pvK} is the probability mass function of output vi∈ V and the channel conditional density p(u|x) satisfies the dominated condition (often true in practice), then the existing SMAWK algorithm can be applied to reduce the time complexity of the dynamic programming algorithm from O(KNM2) to O(KNM). Both theoretical and numerical results are provided to verify our contributions.

Consider a channel having the discrete input X that is corrupted by a continuous noise to produce the continuous-valued output U. A thresholding quantizer is then used to quantize the continuous-valued output U to the final discrete output V . One wants to design a thresholding quantizer that maximizes the mutual information between the input and the final quantized output I(X; V ). In this paper, the structure of optimal thresholding quantizer is established that finally results in two efficient algorithms having the time complexities O(NM + K log2(NM)) for finding the local optimal quantizer and O(K M log(NM)) for finding the global optimal quantizer where N, M, K are the size of input X, received output U and quantized output V , respectively. Both theoretical and numerical results are provided to verify our contributions.

We consider a channel with a binary input X being corrupted by a continuous-valued noise that results in a continuous-valued output Y . An optimal binary quantizer is used to quantize the continuous-valued output Y to the final binary output Z to maximize the mutual information I(X; Z). We show that when the ratio of the channel conditional density r(y) = P (Y =y|X=0)/P (Y =y|X=1) is a strictly increasing or decreasing function of y, then a quantizer having a single threshold can maximize mutual information. Furthermore, we show that an optimal quantizer (possibly with multiple thresholds) is the one with the thresholding vector whose elements are all the solutions of r(y) = r* for some constant r* > 0. In addition, we also characterize necessary conditions using fixed point theorem for the optimality and uniqueness of a quantizer. Based on these conditions, we propose an efficient procedure for determining all locally optimal quantizers, and thus, a globally optimal quantizer can be found. Our results also confirm some previous results using alternative elementary proofs.

In this paper, we study the channel capacity of discrete memoryless thresholding channels (DMTCs) that are used in Pulse Amplitude Modulation (PAM) for LED free-space transmissions. Although capacities of discrete memoryless channels (DMC) are well-studied and can be determined by various algorithms, the capacity of a DMTC is less explored and its capacity is more difficult to obtain. This is due to the fact that, unlike a typical DMC channel whose the capacity is a function input distribution, the capacity of a DTMC channel is a function of both input distribution and decision thresholds. To resolve this problem, we propose an algorithm for finding the channel capacity of a DMTC. Both theoretical and numerical results are provided to verify our approach.

While capacities of discrete memoryless channels are well studied, it is still not possible to obtain a closed form expression of the capacity for an arbitrary discrete memoryless channel. This paper shows an elementary technique based on Karush-Kuhn-Tucker (KKT) conditions to obtain a closed form expression for a good upper bound of an arbitrary discrete memoryless channel. Furthermore, using this technique, we are able to obtain the closed form expressions for the capacities of channels whose channel matrices satisfy a number of conditions.

Free Space Optical (FSO) communication has received tremendous attention in recent years as an alternative approach to overcome the limited RF spectrum problem for wireless communication. Proposed FSO communication systems transmit information by modulating light. Thus, FSO transmissions do not interfere with existing RF transmissions, and can provide substantial capacity gain. In this paper, we briefly introduce WiFO, a hybrid indoor wireless communication system designed to substantially increase wireless capacity. We also describe a new channel model used for multi-user FSO communication. The main contributions of this work are (1) characterizing the multiuser achievable rate region for the new channel given the side information via a constructive embedded coding technique and (2) providing analytical result for maximizing the sum rate of users.

The recent increase in the number of wireless devices has been driven by the growing markets of smart homes and the Internet of Things (IoT). As a result, expanding and/or efficient utilization of the radio frequency (RF) spectrum is critical to accommodate such an increase in wireless bandwidth. Alternatively, recent free-space optical (FSO) communication technologies have demonstrated the feasibility of building WiFO, a high capacity indoor wireless network using the femtocell architecture. Since FSO transmission does not interfere with the RF signals, such a system can be integrated with the current WiFi systems to provide orders of magnitude improvement in bandwidth. A novel component of WiFO is its ability to jointly encode bits from different flows for optimal transmissions. In this paper, we introduce the WiFO architecture and a novel cooperative transmission framework using location assisted coding (LAC) technique to increase the overall wireless capacity. Specifically, achievable rate regions for WiFO using LAC will be characterized. Both numerical and theoretical analyses are given to validate the proposed coding schemes.