Whole Slide Images (WSIs) are high-resolution digital scans widely used in medical diagnostics. Due to their immense size, WSI classification is typically approached using Multiple Instance Learning (MIL), where a slide is partitioned into individual tiles, disrupting its spatial structure. Recent MIL methods often incorporate spatial context through rigid spatial assumptions (e.g. fixed kernels), which limit their ability to capture the intricate tissue structures crucial for an accurate diagnosis. To address this limitation, we propose Probabilistic Spatial Attention MIL (PSA-MIL), a novel attention-based MIL framework that integrates spatial context into the attention mechanism through learnable distance-decayed priors, formulated within a probabilistic interpretation of self-attention as a posterior distribution. This formulation enables a dynamic inference of spatial relationships during training, eliminating the need for predefined assumptions often imposed by previous approaches. Furthermore, we introduce a diversity loss that promotes complementary spatial representations across attention heads and a spatial posterior-pruning strategy that reduces computational cost for long WSI sequences while preserving performance. Extensive experiments across multiple datasets and tasks show that PSA-MIL outperforms current baselines and achieves state-of-the-art results with substantially lower computational overhead. Our code is available at https://github.com/SharonPeled/PSA-MIL.

• Novel cross-cancer framework improves generalization across tumor types. • Cancer-Aware VisionTransformer extracts shared and cancer-specific features. • Adversarial regularization for enhancing universal features. • Hierarchical balancing addresses data imbalances for unbiased learning. • Benchmark dataset combines multiple cancer types to support multi-cancer digital pathology. Whole Slide Images (WSIs) are crucial in modern pathology, offering high-resolution data for accurate diagnosis, treatment planning, and research. Deep learning methods have recently been proposed to harness this data by extracting and interpreting complex patterns. However, these approaches often focus on specific tumor types, limiting their generalizability across diverse pathological conditions and restricting scalability. This relatively narrow focus ultimately stems from the inherent heterogeneity in histopathology and the diverse morphological and molecular characteristics of different tumors. To this end, we propose a novel approach for multi-cancer WSI analysis, designed to leverage the diversity of different tumor types. We introduce a Cancer-Aware Attention module that models both shared patterns across cancers and cancer-specific variations to address heterogeneity and enhance cross-tumor generalization. Furthermore, we construct an adversarial cancer regularization mechanism to minimize cancer-specific biases through mutual information minimization. Additionally, we develop a hierarchical sample balancing strategy to mitigate data imbalances and promote unbiased learning. Together, these form a cohesive framework for unbiased multi-cancer WSI analysis. Extensive experiments on a uniquely constructed multi-cancer dataset demonstrate significant improvements in generalization, providing a scalable solution for WSI classification across diverse cancer types.