This paper proposes a convolutional neural network (CNN)-based detector for faster-than-Nyquist (FTN) signaling that employs structured fixed kernel layers with domain-informed masking to mitigate intersymbol interference (ISI). Unlike standard CNNs with sliding kernels, the proposed method utilizes fixed-position kernels to directly capture ISI effects at varying distances from the central symbol. A hierarchical filter allocation strategy is also introduced, assigning more filters to earlier layers for strong ISI patterns and fewer to later layers for weaker ones. This design improves detection accuracy while reducing redundant operations. Simulation results show that the detector achieves near-optimal bit error rate (BER) performance for τ ≥ 0.7, closely matching the BCJR algorithm, and offers computational gains of up to 46% and 84% over M-BCJR for BPSK and QPSK, respectively. Comparative analysis with other methods further highlights the efficiency and effectiveness of the proposed approach. To the best of our knowledge, this is the first application of a fixed-kernel CNN architecture tailored for FTN detection in the literature.

This paper presents a novel convolutional neural network (CNN)-based detector for faster-than-Nyquist (FTN) signaling, introducing structured fixed kernel layers with domain-informed masking to effectively mitigate intersymbol interference (ISI). Unlike standard CNN architectures that rely on moving kernels, the proposed approach employs fixed convolutional kernels at predefined positions to explicitly learn ISI patterns at varying distances from the central symbol. To enhance feature extraction, a hierarchical filter allocation strategy is employed, assigning more filters to earlier layers for stronger ISI components and fewer to later layers for weaker components. This structured design improves feature representation, eliminates redundant computations, and enhances detection accuracy while maintaining computational efficiency. Simulation results demonstrate that the proposed detector achieves near-optimal bit error rate (BER) performance, comparable to the BCJR algorithm for the compression factor τ ≥ 0.7, while offering up to 46% and 84% computational cost reduction over M-BCJR for BPSK and QPSK, respectively. Additional evaluations confirm the method’s adaptability to high-order modulations (up to 64-QAM), resilience in quasi-static multipath Rayleigh fading channels, and effectiveness under LDPC-coded FTN transmission, highlighting its robustness and practicality.