Recent studies have raised significant concerns regarding the reliability of current mathematical benchmarks, highlighting key limitations such as simplistic design and potential data contamination that undermine evaluation accuracy. Consequently, developing a reliable benchmark that effectively evaluates large language models' (LLMs) genuine capabilities in mathematical reasoning remains a critical challenge. To address these concerns, we propose RV-Bench, a novel evaluation methodology for Benchmarking LLMs with Random Variables in mathematical reasoning. Specifically, we develop question-generating functions to produce random variable questions (RVQs), whose background content mirrors the original benchmark problems, but with randomized variable combinations, rendering them "unseen" to LLMs. Models must completely understand the inherent question pattern to correctly answer RVQs with diverse variable combinations. Thus, an LLMs' genuine reasoning capability is reflected through its accuracy and robustness on RV-Bench. We conducted extensive experiments on over 30 representative LLMs across more than 1,000 RVQs. Our findings reveal that LLMs exhibit a proficiency imbalance between encountered and "unseen" data distributions. Furthermore, RV-Bench reveals that proficiency generalization across similar mathematical reasoning tasks is limited, but we verified that it can still be effectively elicited through test-time scaling.

Chuang Zhou, Zhu Wang, Shengyuan Chen, Jiahe Du, Qiyuan Zheng, Zhaozhuo Xu, Xiao Huang. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Knowledge graph employs the specific graph structure to store knowledge in the form of entities, relations, attributes, and so forth, which can effectively represent correlations among data and has been applied in many fields, including search engine optimization, intelligent question answering, and recommendation systems. In this article, we mainly focus on the research and application of the domain-specific knowledge graph in the field of the smart city, which has not been fully paid attention to. Currently, the major problem faced by the smart city lies in data mining and proper application. On the one hand, data are usually stored by government management departments, which creates challenges such as high data storing overhead and inefficient data usage. On the other hand, data cannot be coordinated and collaborated between different city management systems, because data silos exist. By constructing the corresponding knowledge graph, the data of urban traffic, services, and public resources are integrated to provide help for city builders and managers to make important decisions. Therefore, we will review the related literature on the knowledge graph existing in the smart city domain to expore reasearch scopes. Specifically, we will analyze and summarize knowledge graph construction research in the field of smart cities from four perspectives, i.e., smart city ontology, urban data processing, urban knowledge graph construction, and their application. Finally, the research limitations and prospects of the urban knowledge graph are provided.