Abstract: Background: In present communication systems, the transmission signals are considered to be time-invariant and are processed by the time-invariant bandpass filter at the receiver. In this regard, all the signals in the pass band will pass through the filter, including the interferences. Materials and Methods: This study analyzes the time-frequency characteristic of the 4-QAM modulated pulse shaping signal and reveals that the energy concentration of the pulse shaping signal is time-varying, which makes some low-energy areas exist in the pass-band. Based on this characteristic, the masking threshold constrained time-varying filter is proposed and the optimal masking threshold is determined with MMSE criterion. Results: Simulation results show that based on the signal’s time-frequency distribution, the SINR and BER performances of the proposed filter outperform the conventional bandpass filter. Conclusion: According to the 4-QAM modulated pulse shaping signal’s energy concentration nature, the non-orthogonal or inter-cell interferences could be suppressed further by the masking threshold constrained time-varying filter.