The data-driven time–frequency analysis (TFA) method has garnered widespread attention due to its robust feature learning and representation capabilities. However, existing methods still require further development in characterizing nonstationary signals with closely-spaced and crossing frequencies generated from wind turbines , and realizing mechanical fault detection. To this end, a novel method, termed time–frequency self-similarity enhancement network (TFSSEN), is proposed. First, an adaptive time–frequency characterizing module (ATFCM), consisting of the time–frequency convolutional layer and adaptive convolutional pooling unit, is designed to represent random scale vibration signals to an appropriate scale time–frequency representation (TFR). Second, a non-local and global attention residual group (NGARG) is constructed, where a single-scale self-similarity exploitation module is introduced to calculate feature correlations within single-scale TFR, and an improved-global context attention mechanism is developed to explore the most informative components in multi-scale time–frequency features, thereby achieving precise feature reconstruction. Finally, the self-similarity mixed-scale time–frequency enhancement module (SMTEM) is constructed by multiple cascaded NGARGs, and it can extract frequency information from similar time–frequency features and gradually enhance energy concentration. Simulation results show that the TFSSEN can effectively characterize nonstationary signal with closely-spaced and crossing frequencies. The comprehensive experiment analysis on the wind turbine planetary gearbox and bearings further demonstrates that the TFSSEN exhibits superior performance for characterizing nonstationary fault characteristic frequencies (FCFs) compared with advanced TFA methods.