Off-topic detection analyzes the closeness between a user’s response and a specific topic in a given context or question. The development of off-topic detection techniques stems from the real-world needs of automated scoring systems, as feature-based machine learning methods are only based on shallow features of the text. For example, sentence length, number of words, etc., cannot reflect the relevance of the user’s answer to the topic from the semantic level, resulting in the content of the user’s answer containing some answers that do not have grammatical errors but are not relevant to the topic can also get a higher score, which significantly affects the impartiality and fairness of the automatic scoring system. Therefore, this paper proposes a deep learning model based on the combination of Siamese network and a cross-attention mechanism for the off-topic detection task, which is suitable for automatic short-answer off-topic detection scenarios. By conducting experiments using the publicly available dataset Mohler, the experimental results show that our proposed model is adequate. The experimental results in this paper achieve 91.9% F1 value, 92.5% recall, and 94.1% precision. The F1 value and precision are improved by 3.7% and 3%, respectively, compared to the best-performing baseline model, the Bert-based deep neural network model.