The opaque nature of deep learning models presents significant challenges for the ethical deployment of hate speech detection systems. To address this limitation, we introduce Supervised Rational Attention (SRA), a framework that explicitly aligns model attention with human rationales, improving both interpretability and fairness in hate speech classification. SRA integrates a supervised attention mechanism into transformer-based classifiers, optimizing a joint objective that combines standard classification loss with an alignment loss term that minimizes the discrepancy between attention weights and human-annotated rationales. We evaluated SRA on hate speech benchmarks in English (HateXplain) and Portuguese (HateBRXplain) with rationale annotations. Empirically, SRA achieves 2.4× better explainability compared to current baselines, and produces token-level explanations that are more faithful and human-aligned. In terms of fairness, SRA achieves competitive fairness across all measures, with second-best performance in detecting toxic posts targeting identity groups, while maintaining comparable results on other metrics. These findings demonstrate that incorporating human rationales into attention mechanisms can enhance interpretability and faithfulness without compromising fairness.

We address the challenge of minimizing "true risk" in multi-node distributed learning.\footnote{We use the term node to refer to a client, FPGA, APU, CPU, GPU, or worker.} These systems are frequently exposed to both inter-node and intra-node "label shifts", which present a critical obstacle to effectively optimizing model performance while ensuring that data remains confined to each node. To tackle this, we propose the Versatile Robust Label Shift (VRLS) method, which enhances the maximum likelihood estimation of the test-to-train label importance ratio. VRLS incorporates Shannon entropy-based regularization and adjusts the importance ratio during training to better handle label shifts at the test time. In multi-node learning environments, VRLS further extends its capabilities by learning and adapting importance ratios across nodes, effectively mitigating label shifts and improving overall model performance. Experiments conducted on MNIST, Fashion MNIST, and CIFAR-10 demonstrate the effectiveness of VRLS, outperforming baselines by up to 20\% in imbalanced settings. These results highlight the significant improvements VRLS offers in addressing label shifts. Our theoretical analysis further supports this by establishing high-probability bounds on estimation errors.

Modern deep neural networks exhibit heterogeneity across numerous layers of various types such as residuals, multi-head attention, etc., due to varying structures (dimensions, activation functions, etc.), distinct representation characteristics, which impact predictions. We develop a general layer-wise quantization framework with tight variance and code-length bounds, adapting to the heterogeneities over the course of training. We then apply a new layer-wise quantization technique within distributed variational inequalities (VIs), proposing a novel Quantized Optimistic Dual Averaging (QODA) algorithm with adaptive learning rates, which achieves competitive convergence rates for monotone VIs. We empirically show that QODA achieves up to a $150$% speedup over the baselines in end-to-end training time for training Wasserstein GAN on $12+$ GPUs.

While momentum-based accelerated variants of stochastic gradient descent (SGD) are widely used when training machine learning models, there is little theoretical understanding on the generalization error of such methods. In this work, we first show that there exists a convex loss function for which the stability gap for multiple epochs of SGD with standard heavy-ball momentum (SGDM) becomes unbounded. Then, for smooth Lipschitz loss functions, we analyze a modified momentum-based update rule, i.e., SGD with early momentum (SGDEM) under a broad range of step-sizes, and show that it can train machine learning models for multiple epochs with a guarantee for generalization. Finally, for the special case of strongly convex loss functions, we find a range of momentum such that multiple epochs of standard SGDM, as a special form of SGDEM, also generalizes. Extending our results on generalization, we also develop an upper bound on the expected true risk, in terms of the number of training steps, sample size, and momentum. Our experimental evaluations verify the consistency between the numerical results and our theoretical bounds. SGDEM improves the generalization error of SGDM when training ResNet-18 on ImageNet in practical distributed settings.

We consider device-to-device (D2D) communication underlaid in a cellular network to share the uplink resource of cellular users (CUs). It is a key the emerging Internet of Things to support vehicle-to-everything communication networks. In a multicell scenario, both D2D pairs and CUs may cause significant significant intercell interference (ICI) to the neighboring cells. Furthermore, due to substantial signaling overhead, we assume only partial channel state information (CSI) of D2D links at the base station. We consider joint power control, beamforming, and CU-D2D matching problem, assuming partial CSI from D2D pairs under the general Nakagami fading model. We formulate a joint receive beamforming and robust power control optimization problem for a CU-D2D pair to expected sum rate under the power budget, while meeting the minimum SINR requirements and worst case ICI limits at neighboring cells in a probabilistic sense. We propose an efficient algorithm that combines an iterative D2D feasibility check and a ratio-of-expectation approximation. A performance upper bound is also developed for benchmarking. For multiple CUs and D2D pairs, due to orthogonal channelization within each cell, we first focus on the problem of joint power control and beamforming for a CU-D2D pair and show how our proposed solution can be leveraged to find a solution for this general problem. The complexity analysis of the proposed approach is also provided. Simulation results show that the proposed algorithm gives performance close to the upper bound.

Overparameterization refers to the important phenomenon where the width of a neural network is chosen such that learning algorithms can provably attain zero loss in nonconvex training. The existing theory establishes such global convergence using various initialization strategies, training modifications, and width scalings. In particular, the state-of-the-art results require the width to scale quadratically with the number of training data under standard initialization strategies used in practice for best generalization performance. In contrast, the most recent results obtain linear scaling either with requiring initializations that lead to the "lazy-training", or training only a single layer. In this work, we provide an analytical framework that allows us to adopt standard initialization strategies, possibly avoid lazy training, and train all layers simultaneously in basic shallow neural networks while attaining a desirable subquadratic scaling on the network width. We achieve the desiderata via Polyak-Lojasiewicz condition, smoothness, and standard assumptions on data, and use tools from random matrix theory.

As the size and complexity of models and datasets grow, so does the need for communication-efficient variants of stochastic gradient descent that can be deployed to perform parallel model training. One popular communication-compression method for data-parallel SGD is QSGD (Alistarh et al., 2017), which quantizes and encodes gradients to reduce communication costs. The baseline variant of QSGD provides strong theoretical guarantees, however, for practical purposes, the authors proposed a heuristic variant which we call QSGDinf, which demonstrated impressive empirical gains for distributed training of large neural networks. In this paper, we build on this work to propose a new gradient quantization scheme, and show that it has both stronger theoretical guarantees than QSGD, and matches and exceeds the empirical performance of the QSGDinf heuristic and of other compression methods.

Many communication-efficient variants of SGD use gradient quantization schemes. These schemes are often heuristic and fixed over the course of training. We empirically observe that the statistics of gradients of deep models change during the training. Motivated by this observation, we introduce two adaptive quantization schemes, ALQ and AMQ. In both schemes, processors update their compression schemes in parallel by efficiently computing sufficient statistics of a parametric distribution. We improve the validation accuracy by almost 2% on CIFAR-10 and 1% on ImageNet in challenging low-cost communication setups. Our adaptive methods are also significantly more robust to the choice of hyperparameters.

For device-to-device (D2D) underlaid cellular networks, the perfect channel state information (CSI) may not be available at the base station (BS). In this work, under an assumption of partial CSI, we study the problem of maximizing the expected sum rate for a cellular user (CU) and a D2D pair, with receive beamforming at the BS, subject to minimum SINR requirements for both the CU and D2D pair, per-node maximum power, and inter-cell interference constraints in multiple neighboring cells. We solve this non-convex joint optimization problem in two steps. We first consider the D2D admissibility problem to determine whether the D2D pair can reuse the channel resource of the CU. We then propose a robust power control algorithm using a ratio-of-expectation approximation to maximize the expected sum rate. For benchmarking, we further provide an upper bound on the maximum expected sum rate. Simulation results show that our proposed solution gives performance close to the upper bound.

For device-to-device (D2D) communication under laid in a cellular network with uplink resource sharing, both cellular and D2D pairs may cause significant inter-cell interference (ICI) at a neighboring base station (BS). In this paper, under optimal BS receive beamforming, we jointly optimize the power of a cellular user (CU) and a D2D pair for their sum rate maximization, while satisfying minimum SINR requirements and worst-case ICI limit in multiple neighboring cells. We solve this non-convex joint optimization problem in two steps. First, the necessary and sufficient condition for the D2D admissibility under given constraints is obtained. Finally, we consider joint power control of the CU and D2D transmitters. We propose a power control algorithm to maximize the sum rate. Depending on the severity of ICI that D2D and CU may cause, we categorize the feasible solution region into five cases, each of which may further include several scenarios based on minimum SINR requirements. The proposed algorithm is optimal when ICI to a single neighboring cell is considered. For multiple neighboring cells, we provide an upper bound on the performance loss by the proposed algorithm and conditions for its optimality. We further extend our consideration to the scenario of multiple CUs and D2D pairs, and formulate the joint power control and CU-D2D matching problem. We show how our proposed solution for one CU and one D2D pair can be utilized to solve this general joint optimization problem. Simulation demonstrates the effectiveness of our power control algorithm and the nearly optimal performance of the proposed approach in the setting of multiple CUs and D2D pairs.

We consider a cellular network where each cell contains multiple source-destination pairs communicating through multiple amplify-and-forward relays using orthogonal channels. We propose an optimal relay beamforming design that minimizes the maximum interference at the neighboring cells subject to per-relay power limits and minimum received signal-to-noise ratio (SNR) requirements. Even though the problem is non-convex, we show that it has zero Lagrange duality gap, and we convert its dual problem to a semi-definite programming problem. Depending on the values of the optimal dual variables, we study three cases to obtain the optimal beam vectors accordingly. This results in an iterative algorithm that provides a semi-closed-form optimal solution. We extend our algorithm to the problem of maximizing the minimum SNR subject to some pre-determined maximum interference constraints at neighboring cells, by the solution to the min-max interference problem along with a bisection search. The solution to this max-min SNR problem gives insight into the worst-case signal-to-interference-and-noise ratio given some maximum interference target. The performance of the proposed algorithm is studied numerically, both for when the knowledge of interference channel is perfect and for when it is imperfect due to either limited feedback or channel estimation error.

We investigate the optimal relay beamforming problem for multi-user peer-to-peer communication with amplify-and-forward relaying in a multi-channel system. Assuming each source-destination (S-D) pair is assigned an orthogonal channel, we formulate the problem as a min-max per-relay power minimization problem with minimum signal-to-noise (SNR) guarantees. After showing that strong Lagrange duality holds for this nonconvex problem, we transform its Lagrange dual problem to a semi-definite programming problem and obtain the optimal relay beamforming vectors. We identify that the optimal solution can be obtained in three cases, depending on the values of the optimal dual variables. These cases correspond to whether the minimum SNR requirement at each S-D pair is met with equality, and whether the power consumption at a relay is the maximum among relays at optimality. We obtain a semi-closed form solution structure of relay beam vectors, and propose an iterative approach to determine relay beam vector for each S-D pair. We further show that the reverse problem of maximizing the minimum SNR with per-relay power budgets can be solved using our proposed algorithm with an iterative bisection search. Through simulation, we analyze the effect of various system parameters on the performance of the optimal solution. Furthermore, we investigated the effect of imperfect channel side information of the second hop on the performance and quantify the performance loss due to either channel estimation error or limited feedback.

We consider a wireless cellular network with multiple amplify-and-forward (AF) relays in each cell, assisting the communication of multiple source-destination pairs with relay transmission beamforming. Our objective is to minimize the maximum interference power among all active receivers in a neighboring cell subject to per-relay power and minimum received SNR constraints. We propose an efficient algorithm to obtain the optimal relay beamforming vectors. We show that even though the optimization problem is non-convex, it has zero Lagrange duality gap and can be converted to a semi-definite programming problem. The performance of the proposed algorithm is studied numerically, both for the case where the interference channel information is exactly known and for the case of inaccurate channel information due to either limited feedback or channel estimation error. It is demonstrated that the min-max interference approach substantially outperforms the alternative where we simply minimize the maximum relay transmission power.

Likelihood-based algorithms for the classification of linear digital modulations are systematically investigated for a multiple receive antennas configuration. Existing modulation classification (MC) algorithms are first extended to the case of multiple receive antennas and then a critical problem is identified that the overall performance of the multiple antenna systems is dominated by the worst channel estimate of a particular antenna. To address the performance degradation issue, we propose a new MC algorithm by optimally combining the log likelihood functions (LLFs). Furthermore, to analyze the upper-bound performance of the existing and the proposed MC algorithms, the exact Cramer-Rao Lower Bound (CRLB) expressions of non-data-aided joint estimates of amplitude, phase, and noise variance are derived for general rectangular quadrature amplitude modulation (QAM). Numerical results demonstrate the accuracy of the CRLB expressions and verify that the results reported in the literature for quadrature phase-shift keying (QPSK) and 16-QAM are special cases of our derived expressions. Also, it is demonstrated that the probability of correct classification of the new algorithm approaches the theoretical bounds and a substantial performance improvement is achieved compared to the existing MC algorithm.