Supervised learning with tabular data presents unique challenges, including low data sizes, the absence of structural cues, and heterogeneous features spanning both categorical and continuous domains. Unlike vision and language tasks, where models can exploit inductive biases in the data, tabular data lacks inherent positional structure, hindering the effectiveness of self-attention mechanisms. While recent transformer-based models like TabTransformer, SAINT, and FT-Transformer (which we refer to as 3T) have shown promise on tabular data, they typically operate without leveraging structural cues such as positional encodings (PEs), as no prior structural information is usually available. In this work, we find both theoretically and empirically that structural cues, specifically PEs can be a useful tool to improve generalization performance for tabular transformers. We find that PEs impart the ability to reduce the effective rank (a form of intrinsic dimensionality) of the features, effectively simplifying the task by reducing the dimensionality of the problem, yielding improved generalization. To that end, we propose Tab-PET (PEs for Tabular Transformers), a graph-based framework for estimating and inculcating PEs into embeddings. Inspired by approaches that derive PEs from graph topology, we explore two paradigms for graph estimation: association-based and causality-based. We empirically demonstrate that graph-derived PEs significantly improve performance across 50 classification and regression datasets for 3T. Notably, association-based graphs consistently yield more stable and pronounced gains compared to causality-driven ones. Our work highlights an unexpected role of PEs in tabular transformers, revealing how they can be harnessed to improve generalization.

Estimating the confidence of deep neural network predictions is crucial for safe deployment in high-stakes applications. While softmax probabilities are commonly used, they are often poorly calibrated, and existing calibration methods have been shown to be detrimental to failure prediction. In this paper, we propose using information-theoretic measures to estimate prediction confidence in a post-hoc manner, without modifying network architecture or training. Specifically, we compare three pointwise information (PI) measures: pointwise mutual information (PMI), pointwise $\mathcal{V}$-information (PVI), and the recently proposed pointwise sliced mutual information (PSI). These measures are theoretically grounded in their relevance to predictive uncertainty, with properties such as invariance, convergence rates, and sensitivity to geometric attributes like margin and intrinsic dimensionality. Through extensive experiments on benchmark computer vision models and datasets, we find that PVI consistently outperforms PMI, PSI and existing post-hoc baselines in failure prediction across metrics. For confidence calibration, PVI matches the performance of temperature-scaled softmax, which is already regarded as a highly effective baseline. This indicates that PVI achieves superior failure prediction without compromising its calibration performance. This aligns with our theoretical insights, which suggest that PVI offers the most balanced trade-offs.

Conventional communication systems focus on re-covering the messages sent by the transmitter at the receiver, by undoing the errors introduced by the channel. In semantic communication, the goal is instead to preserve the semantics (i.e., meaning) of the intended message$X$. A well-known framework adopted by the literature in modelling semantic communication is task-oriented communication. There, an encoder generates a feature representation$Z$(e.g., a trained neural network) to transmit across the channel to fulfill a certain task (e.g., classification with labels$Y$). This naturally leads to the study of the information bottleneck (IB) principle - which proposes reducing the mutual information$I (X; Z)$(compression) while simultaneously maxi-mizing$I (Z; Y)$(fitting). Although this objective seems initially meaningful, we posit that both the standard task-oriented setting for studying semantic communication and the IB approach may not be optimal. In this paper, we first propose a novel paradigm of multi-task generalizable communication, where, at test time, the messages may originate from a different unseen classification task compared to those during training. We intuitively reason that the multi-task generalizable setting is more feasible and relevant in the context of real-world semantic communication, compared to task-oriented communication. Next, we propose a Reconstruction Loss Aware (RLA) approach, which yields better feature encodings that are more generalizable towards unseen tasks. We empirically observe that, in the proposed paradigm, the IB principle is not optimal. We also demonstrate that RLA is better (15%-20% higher accuracy) compared to a benchmark approach for task-oriented communication, for diverse non-overlapping unseen tasks.

Most entropy measures depend on the spread of the probability distribution over the sample space |X|, and the maximum entropy achievable scales proportionately with the sample space cardinality |X|. For a finite |X|, this yields robust entropy measures which satisfy many important properties, such as invariance to bijections, while the same is not true for continuous spaces (where |X|=infinity). Furthermore, since R and R^d (d in Z+) have the same cardinality (from Cantor's correspondence argument), cardinality-dependent entropy measures cannot encode the data dimensionality. In this work, we question the role of cardinality and distribution spread in defining entropy measures for continuous spaces, which can undergo multiple rounds of transformations and distortions, e.g., in neural networks. We find that the average value of the local intrinsic dimension of a distribution, denoted as ID-Entropy, can serve as a robust entropy measure for continuous spaces, while capturing the data dimensionality. We find that ID-Entropy satisfies many desirable properties and can be extended to conditional entropy, joint entropy and mutual-information variants. ID-Entropy also yields new information bottleneck principles and also links to causality. In the context of deep learning, for feedforward architectures, we show, theoretically and empirically, that the ID-Entropy of a hidden layer directly controls the generalization gap for both classifiers and auto-encoders, when the target function is Lipschitz continuous. Our work primarily shows that, for continuous spaces, taking a structural rather than a statistical approach yields entropy measures which preserve intrinsic data dimensionality, while being relevant for studying various architectures.

In this paper, we study the memorization and generalization behaviour of deep neural networks (DNNs) using sliced mutual information (SMI), which is the average of the mutual information (MI) between one-dimensional random projections. We argue that the SMI between features in a DNN ($T$) and ground truth labels ($Y$), $SI(T;Y)$, can be seen as a form of usable information that the features contain about the labels. We show theoretically that $SI(T;Y)$ can encode geometric properties of the feature distribution, such as its spherical soft-margin and intrinsic dimensionality, in a way that MI cannot. Additionally, we present empirical evidence showing how $SI(T;Y)$ can capture memorization and generalization in DNNs. In particular, we find that, in the presence of label noise, all layers start to memorize but the earlier layers stabilize more quickly than the deeper layers. Finally, we point out that, in the context of Bayesian Neural Networks, the SMI between the penultimate layer and the output represents the worst case uncertainty of the network’s output.

When deploying deep learning models such as convolutional neural networks (CNNs) in safety-critical domains, it is important to understand the predictions made by these black-box models. While many different approaches have been taken to improve the interpretability of deep models, most methods lack theoretical properties, making it hard to justify the correctness of the explanations provided. In this paper, we take an information-theoretic approach to understand why CNNs make their predictions. Building upon sliced mutual information, we propose pointwise sliced mutual information (PSI) as a tool for measuring the amount of useful information that a feature has about the label for a single instance. We theoretically justify the use of PSI for explaining predictions made by CNNs through the connection with margin. We show that PSI works as an explainability tool in two ways: (i) Fiber-wise PSI constructs a saliency map that highlights regions in the image which are important in predicting the labels; (ii) Sample-wise PSI provides confidence scores for predictions on the labels.

Along with the practical success of deep neural networks, several theories have been proposed to explain their excellent generalization behaviour. One such theory is information bottleneck, using mutual information (MI) as a measure to understand the learning dynamics of these black-box models. However, estimating MI in high dimensions and in deterministic settings is problematic, often resulting in widely varying estimates for different estimators. This paper takes an alternative approach to analyze the behaviour of deep models by using a recently proposed information measure known as sliced mutual information (SMI), which is more computationally efficient to estimate than MI in high dimensions. We theoretically connect SMI to the classifier margin, thereby showcasing its ability to encode geometric properties of the feature distribution. We also study SMI empirically and demonstrate that the SMI between the hidden layer and the labels encodes information about the network’s ability to predict labels correctly.

What makes a classifier have the ability to generalize? There have been a lot of important attempts to address this question, but a clear answer is still elusive. Proponents of complexity theory find that the complexity of the classifier's function space is key to deciding generalization, whereas other recent work reveals that classifiers which extract invariant feature representations are likely to generalize better. Recent theoretical and empirical studies, however, have shown that even within a classifier's function space, there can be significant differences in the ability to generalize. Specifically, empirical studies have shown that among functions which have a good training data fit, functions with lower Kolmogorov complexity (KC) are likely to generalize better, while the opposite is true for functions of higher KC. Motivated by these findings, we propose, in this work, a novel measure of complexity called Kolmogorov Growth (KG), which we use to derive new generalization error bounds that only depend on the final choice of the classification function. Guided by the bounds, we propose a novel way of regularizing neural networks by constraining the network trajectory to remain in the low KG zone during training. Minimizing KG while learning is akin to applying the Occam's razor to neural networks. The proposed approach, called network-to-network regularization, leads to clear improvements in the generalization ability of classifiers. We verify this for three popular image datasets (MNIST, CIFAR-10, CIFAR-100) across varying training data sizes. Empirical studies find that conventional training of neural networks, unlike network-to-network regularization, leads to networks of high KG and lower test accuracies. Furthermore, we present the benefits of N2N regularization in the scenario where the training data labels are noisy. Using N2N regularization, we achieve competitive performance on MNIST, CIFAR-10 and CIFAR-100 datasets with corrupted training labels, significantly improving network performance compared to standard cross-entropy baselines in most cases. These findings illustrate the many benefits obtained from imposing a function complexity prior like Kolmogorov Growth during the training process.

This paper describes a novel approach to the problem of autonomous Robot Navigation in environments having less or no source of illumination. We have aimed at depth estimation and object recognition aspects, using the bio-inspired Dynamic Vision Sensor (DVS) asynchronous time-based image sensor (ATIS) silicon retina. Experiments were conducted in a dark environment using the ATIS camera, coupled with a simple point-like white LED light source mounted on the same. Switching the LED on for a fraction of time in the dark environment produced a diverging ripple of events in the ATIS. We show how this event response can be used to quantify the distance of the planar obstacle from the camera and also to characterize the object for use in object recognition. The ripple effect observed can be attributed to the high temporal resolution of the ATIS retina, the small rise time of the LED and the light intensity profile on the wall. In the initial sections of the paper, we have shown the theoretical basis for the phenomenon observed and then moved on to describe the proof of concept for depth estimation and object recognition. The algorithms can be used in robotic systems mounted with the ATIS and LED for real time depth perception and object recognition.