Sedentary behavior (SB) is a major global health concern, necessitating accurate physical activity (PA) intensity monitoring. Conventional machine-learning (ML) methods using accelerometers struggle to generalize due to variability across populations, sensors, and activities, leading to inconsistent real-world performance. This study presents XNet, a dual-domain deep learning (DL) model for classifying PA intensity and estimating energy expenditure. XNet features a hierarchical multihead architecture that independently extracts temporal and frequency features from multiple sensors, then integrates them via a novel attentional feature fusion (AFF) module applied in two stages: first aggregating sensor features, then fusing domain embeddings. This hierarchical approach outperforms single-stage fusion and provides interpretable attention weights revealing sensor and domain contributions. Frequency-domain features are essential for generalization: in cross-dataset evaluations, XNet achieved the highest F1-score of 70.5 while maintaining robust sedentary detection (88% TPR), and in open-set scenarios, it achieved an F1-score of 77.0, surpassing all DL and hand-crafted baselines. We validated XNet on multiple public datasets and a new dataset of 105 participants. Furthermore, our analysis shows that lightweight 1D-convolutional spectral encoders yield better out-of-distribution generalization than transformer and graph attention (GAT) network alternatives, while benchmarking confirms that AFF outperforms nine fusion strategies in balancing accuracy, efficiency, and robustness to sensor failure. The model adapts to physiological signals (heart rate and ECG) and exhibits low inference latency (~25 ms), making it suitable for on-device deployment. A complementary visual analytics framework uses attention weights to facilitate expert auditing, thereby promoting transparent and equitable health monitoring.