The　data-driven　time–frequency　analysis　(TFA)　method　has　garnered　widespread　attention　due　to　its　robust　feature　learning　and　representation　capabilities.　However,　existing　methods　still　require　further　development　in　characterizing　nonstationary　signals　with　closely-spaced　and　crossing　frequencies　generated　from　wind　turbines,　and　realizing　mechanical　fault　detection.　To　this　end,　a　novel　method,　termed　time–frequency　self-similarity　enhancement　network　(TFSSEN),　is　proposed.　First,　an　adaptive　time–frequency　characterizing　module　(ATFCM),　consisting　of　the　time–frequency　convolutional　layer　and　adaptive　convolutional　pooling　unit,　is　designed　to　represent　random　scale　vibration　signals　to　an　appropriate　scale　time–frequency　representation　(TFR).　Second,　a　non-local　and　global　attention　residual　group　(NGARG)　is　constructed,　where　a　single-scale　self-similarity　exploitation　module　is　introduced　to　calculate　feature　correlations　within　single-scale　TFR,　and　an　improved-global　context　attention　mechanism　is　developed　to　explore　the　most　informative　components　in　multi-scale　time–frequency　features,　thereby　achieving　precise　feature　reconstruction.　Finally,　the　self-similarity　mixed-scale　time–frequency　enhancement　module　(SMTEM)　is　constructed　by　multiple　cascaded　NGARGs,　and　it　can　extract　frequency　information　from　similar　time–frequency　features　and　gradually　enhance　energy　concentration.　Simulation　results　show　that　the　TFSSEN　can　effectively　characterize　nonstationary　signal　with　closely-spaced　and　crossing　frequencies.　The　comprehensive　experiment　analysis　on　the　wind　turbine　planetary　gearbox　and　bearings　further　demonstrates　that　the　TFSSEN　exhibits　superior　performance　for　characterizing　nonstationary　fault　characteristic　frequencies　(FCFs)　compared　with　advanced　TFA　methods.　©　2025　Elsevier　Ltd