Gesture recognition provides an effective human-computer interaction that makes device control more intuitive and convenient. Although the research on mmWave radar-based gesture recognition has demonstrated promising results, existing studies have exclusively addressed the cross-domain challenge or the human activity interference problem, and no attention has been paid to the cross-domain problem in the presence of human activity interference. To address these issues, we propose a novel mmWave radar-based gesture recognition system, named GestSAM, which leverages a parameter-free attention mechanism to effectively extract gesture features that are less affected by environmental noise. By integrating this mechanism with deep learning techniques, GestSAM significantly reduces the impact of human activity interference while maintaining robust cross-domain gesture recognition performance. This approach ensures robust, high-accuracy recognition of gestures. In order to evaluate the performance of our system, we construct a dataset containing six different gesture types performed by fifteen volunteers in seven different scenarios and simulate three interference conditions. The experimental results show that under human activity interference, the model achieves average recognition accuracies of 92.79% and 94.62% in cross-user and cross-scenario, respectively.