How to better evaluate the capabilities of Large Language Models (LLMs) is the focal point and hot topic in current LLMs research. Previous work has noted that due to the extremely high cost of iterative updates of LLMs, they are often unable to answer the latest dynamic questions well. To promote the improvement of Chinese LLMs’ ability to answer dynamic questions, in this paper, we introduce CDQA, a Chinese Dynamic QA benchmark containing question-answer pairs related to the latest news on the Chinese Internet. We obtain high-quality data through a pipeline that combines humans and models, and carefully classify the samples according to the frequency of answer changes to facilitate a more fine-grained observation of LLMs’ capabilities. We have also evaluated and analyzed mainstream and advanced Chinese LLMs on CDQA. Extensive experiments and valuable insights suggest that our proposed CDQA is challenging and worthy of more further study. We believe that the benchmark we provide will become one of the key data resources for improving LLMs’ Chinese question-answering ability in the future.

Leveraging mathematical Large Language Models (LLMs) for proof generation is a fundamental topic in LLMs research. We argue that the ability of current LLMs to prove statements largely depends on whether they have encountered the relevant proof process during training. This reliance limits their deeper understanding of mathematical theorems and related concepts. Inspired by the pedagogical method of "proof by counterexamples" commonly used in human mathematics education, our work aims to enhance LLMs’ ability to conduct mathematical reasoning and proof through counterexamples. Specifically, we manually create a high-quality, university-level mathematical benchmark, COUNTERMATH, which requires LLMs to prove mathematical statements by providing counterexamples, thereby assessing their grasp of mathematical concepts. Additionally, we develop a data engineering framework to automatically obtain training data for further model improvement. Extensive experiments and detailed analyses demonstrate that COUNTERMATH is challenging, indicating that LLMs, such as OpenAI o1, have insufficient counterexample-driven proof capabilities. Moreover, our exploration into model training reveals that strengthening LLMs’ counterexample-driven conceptual reasoning abilities is crucial for improving their overall mathematical capabilities. We believe that our work offers new perspectives on the community of mathematical LLMs.

Zhikun Xu, Ming Shen, Jacob Dineen, Zhaonan Li, Xiao Ye, Shijie Lu, Aswin Rrv, Chitta Baral, Ben Zhou. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

Large Language Models (LLMs) have demonstrated outstanding performance in mathematical reasoning capabilities. However, we argue that current large-scale reasoning models primarily rely on scaling up training datasets with diverse mathematical problems and long thinking chains, which raises questions about whether LLMs genuinely acquire mathematical concepts and reasoning principles or merely remember the training data. In contrast, humans tend to break down complex problems into multiple fundamental atomic capabilities. Inspired by this, we propose a new paradigm for evaluating mathematical atomic capabilities. Our work categorizes atomic abilities into two dimensions: (1) field-specific abilities across four major mathematical fields, algebra, geometry, analysis, and topology, and (2) logical abilities at different levels, including conceptual understanding, forward multi-step reasoning with formal math language, and counterexample-driven backward reasoning. We propose corresponding training and evaluation datasets for each atomic capability unit, and conduct extensive experiments about how different atomic capabilities influence others, to explore the strategies to elicit the required specific atomic capability. Evaluation and experimental results on advanced models show many interesting discoveries and inspirations about the different performances of models on various atomic capabilities and the interactions between atomic capabilities. Our findings highlight the importance of decoupling mathematical intelligence into atomic components, providing new insights into model cognition and guiding the development of training strategies toward a more efficient, transferable, and cognitively grounded paradigm of "atomic thinking".

This study focuses on the evaluation of the Open Question Answering (Open-QA) task, which can directly estimate the factuality of large language models (LLMs). Current automatic evaluation methods have shown limitations, indicating that human evaluation still remains the most reliable approach. We introduce a new task, QA Evaluation (QA-Eval) and the corresponding dataset EVOUNA, designed to assess the accuracy of AI-generated answers in relation to standard answers within Open-QA. Our evaluation of these methods utilizes human-annotated results to measure their performance. Specifically, the work investigates methods that show high correlation with human evaluations, deeming them more reliable. We also discuss the pitfalls of current methods and methods to improve LLM-based evaluators. We believe this new QA-Eval task and corresponding dataset EVOUNA will facilitate the development of more effective automatic evaluation tools and prove valuable for future research in this area. All resources are available at https://github.com/wangcunxiang/QA-Eval and it is under the Apache-2.0 License.

In this paper, we analyze the spectral efficiency (SE) and energy efficiency (EE) of multi-user (MU) multiple-input and multiple-output (MIMO) in multi-cell downlink networks. We first analyze the achievable sum-rate, i.e., the SE, of MU-MIMO systems with maximal ratio transmission (MRT) and zero-forcing (ZF) precoders in downlink cellular networks. Different from the conventional analysis, we derive the achievable sum-rate under the assumption that the number of the BS antennas is huge but limited. Based on the analytical results, we obtain the system EE and further analyze the effect of the number of BS antennas and scheduled users on system EE. The computer simulation results show that the analytical results is accurate and there exists an optimal relationship between the BS antennas and the scheduled users.

5G network is anticipated to meet the challenging requirements of mobile traffic in the 2020's, which are characterized by super high data rate, low latency, high mobility, high energy efficiency, and high traffic density. This paper provides an overview of China Mobile's 5G vision and potential solutions. Targeting a paradigm shift to user-centric network operation from the traditional cell-centric operation, 5G radio access network (RAN) design considerations are presented, including RAN restructure, Turbo charged edge, core network (CN) and RAN function repartition, and network slice as a service. Adaptive multiple connections in the user-centric operation is further investigated, where the decoupled downlink and uplink, decoupled control and data, and adaptive multiple connections provide sufficient means to achieve a 5G network with “no more cells.” Software-defined air interface (SDAI) is presented under a unified framework, in which the frame structure, waveform, multiple access, duplex mode, and antenna configuration can be adaptively configured. New paradigm of 5G network featuring user-centric network (UCN) and SDAI is needed to meet the diverse yet extremely stringent requirements across the broad scope of 5G scenarios.

Hybrid beamforming has drawn attention with the increasing concern on high frequency band communication and the appearance of flexible structures of base stations. This paper considers the practical transmitter structure that each antenna is only connected to a unique RF chain and optimizes the analog and digital beamforming matrices to maximize the achievable rate with transmit power constraint. Following alternating optimization principle, closed-form relationships between analog and digital precoders are obtained when both amplitude and phase or only phase can be adjusted to form analog beams. Simulation results demonstrate the advantage of our proposed methods over the existing beam steering method in terms of achievable rate with different scale antenna array. This reveals that the method can be applied to both low and high frequency band communications. When increasing the number of propagation paths, contrast to the traditional beam steering method, the performance of proposed methods becomes better while the complexity almost keeps at an acceptable constant level.

In this paper, we study base station (BS) antenna configuration for downlink multi-cell multi-user distributed multiple input multiple output (MIMO) system to maximize its energy efficiency (EE). In order to obtain the optimal configuration of antennas and users, we first derive the EE expression by considering a realistic power consumption model (PCM). Then an energy-efficient optimization model with respect to the number N of radio access unit (RAU) antennas and the number K of users is formulated to obtain the optimal number Nopt of RAU antennas and the optimal number Kopt of users for maximizing the EE. The EE is concave with respect to the number of RAU antennas and users, so an iteration algorithm is proposed to obtain the optimization solution. The simulation results suggest that the proposed algorithm can obtain the optimal configuration of RAU antennas and supportable users and each user needs only about 2 BS antennas to maximize the EE with zero-forcing (ZF) precoding. Moreover, the proposed antenna configuration scheme can obviously improve the EE of distributed MIMO system.

In this paper, we investigate the joint impact of both the number of small BSs and the transmit powers on the energy efficiency (EE) of Heterogeneous cellular networks (HetNets). Based on a typical HetNet structure, we propose a feasible focusing searching algorithm to find the optimal number of small BSs and the corresponding transmit power under coverage and spectral efficiency constraints that maximize the network EE. Simulation results show that there exists an optimal number of small BSs and the transmit power which can not only maximize network EE but also satisfy the constraints.

Non-orthogonal multiple access (NOMA) is expected to be a promising technique for future wireless networks due to its superior spectral efficiency. In this paper, the sum rate optimization problem for multiple-input multiple-output (MIMO) NOMA systems is studied with the total transmit power constraint and the minimum rate constraint of weak user. We first derive a channel state information (CSI) condition in which MIMO NOMA systems can achieve full rate transmission, i.e. the transmission rate of the weak user equals to the channel capacity of weak user. Based on the CSI condition, we propose an optimal power allocation scheme for MIMO NOMA systems, which can achieve the capacity region of MIMO Broadcast channel as dirty paper coding. A low complexity suboptimal scheme is proposed as well for all CSI channel conditions. Numerical results show that the proposed NOMA schemes significantly outperform the traditional time division based single user MIMO scheme and the multi-user MIMO scheme.

Energy efficiency (EE) has become an important design goal for mobile communication systems. Since the EE of a system is evaluated during a period of time, a transmission strategy designed from maximizing instantaneous EE (INEE) may not achieve the maximal EE of the system. To exploit the accumulative nature of the energy, accumulative EE (ACEE) can be used as the objective function, which is the ratio of the accumulated amount of data transmitted to the accumulated energy consumed until the time for optimization. In this paper, we study when ACEE is beneficial to EE-oriented optimization. By taking a single user multi-antenna multi-subcarrier system serving two classes of traffic as an example, we formulate three problems to optimize rate allocation among subcarriers and time slots respectively maximizing the INEE, ACEE and EE upper-bound, which can be easily extended to multi-user systems. We proceed to analyze the behavior and performance of the corresponding solutions. Analytical and simulation results show that using ACEE yields a more energy efficient design for the systems serving best effort traffic with less transmit antennas, serving less users simultaneously or at low signal to noise ratio under time-varying channels. However, the conclusions for real-time traffic are different.

Mobile traffic is projected to increase 1000 times from 2010 to 2020. This poses significant challenges on the 5thgeneration (5G) wireless communication system design, including network structure, air interface, key transmission schemes, multiple access, and duplexing schemes. In this paper, full duplex (FD) is discussed, aiming to highlight some insights on various issues including deployment scenarios, frame structures, reference signals (RS), interference mitigation, transceiver structures/calibration, and extension of time division duplex (TDD)/frequency division duplex (FDD) to full duplex. It is anticipated that with future standardization and deployment of FD systems, TDD and FDD will be harmoniously integrated, supporting all the existing half duplex mobile phones efficiently, and leading to a much enhanced 5G system performance.

Energy efficiency (EE) has become an important design goal for wireless systems. Since the EE of a system is evaluated in a duration where the channel may vary, designing a transmission strategy to maximize instantaneous EE may lead to a loss in achievable EE of the system. To exploit the accumulative information of throughput and energy, accumulative EE (ACEE) can be used as the objective function, which is the ratio of the accumulated throughput divided by the overall energy consumed over the past time slots. By analyzing the solutions of three EE maximal problems to optimize rate allocation among multiple subcarriers in multiple time slots, we show when and why the ACEE can achieve high system EE. Simulation results verify our theoretical analysis. Our analysis show that the ACEE is beneficial for the systems with low circuit power consumption under time-varying channels when either the data rate requirement or signal to noise ratio is low, and either the number of antennas or subcarriers is small.

Energy efficiency (EE) has been recognized as another important metric as well as spectral efficiency (SE). This paper investigates the fundamental properties of the EE-SE relationship. The point-to-point MIMO transmission link is considered with different channel state information (CSI) conditions. Our analysis proves that the slope of EE-SE curves only depends on the circuit power and bandwidth when the SE approaches zero while it depends on the multiplexing gain when the SE approaches infinity. Reducing circuit power is helpful to improve EE in the low SE region while introducing more antennas benefits the EE in the high SE region. We also study the properties of EE-optimal points. It is found that reducing circuit power brings exponential EE increasing with linear SE loss at the EE-optimal points. More multiplexing gain will alleviate the speed of EE reduction with the increase of SE. More diversity gain brings a fixed EE gain.

Shutdown of low traffic load base stations (BSs) is recognized as a promising approach to increase energy-efficiency (EE) and reduce total power consumption, especially for small cell networks (SCN). In this paper, we study BS closing strategies for downlink multi-antenna multi-carrier SCN supporting best effort traffics. We formulate the optimization problem of long-term BS closing, BS-user association and subcarrier allocation to maximize EE or minimize total power consumption under the constraints of average sum rate and rate proportion. We obtain an optimal solution for maximizing EE and a suboptimal solution for minimizing total power consumption. Simulation results show that the solutions provide substantial gain both in saving power consumption and increasing EE, and minimizing the total power consumption will not lead to the maximal EE.

In this paper, we investigate an energy-efficient coordinated multiple point (CoMP) transmission strategy for downlink heterogeneous cellular networks. We combine CoMP joint processing (CoMP-JP) and coordinated beamforming (CoMP-CB), two special cases of CoMP, in a time division manner to improve both energy efficiency (EE) and spectral efficiency (SE). We formulate the problem as minimizing the total transmit power consumed by both the macro- and pico-base stations (BSs) under the constraints on the data rate requirements from the macro- and pico-users, and on the maximum transmit powers of the macro- and pico-BSs. Both the transmit time and the transmit powers allocated to the CoMP-JP and CoMP-CB transmissions are optimized. Simulation results show that the hybrid CoMP-JP and CoMP-CB strategy provides a larger capacity region than the CoMP-JP-only or CoMP-CB-only transmission. The time proportion of the CoMP-JP in the hybrid strategy decreases with the data rate requirement of the macro-user and increases with the maximum transmit power of the pico-BS and the average channel gain from the macro-BS to the macro-user. Increasing the transmit power of the pico-BS can improve the EE in the high SE region of the macro-user.

In this paper, we investigate adaptive configuration of spatial and frequency resources to maximize energy efficiency (EE) and reveal the relationship between the EE and the spectral efficiency (SE) in downlink multiple-input-multiple-output (MIMO) orthogonal frequency division multiple access (OFDMA) systems. We formulate the problem as minimizing the total power consumed at the base station under constraints on the ergodic capacities from multiple users, the total number of subcarriers, and the number of radio frequency (RF) chains. A three-step searching algorithm is developed to solve this problem. We then analyze the impact of spatial-frequency resources, overall SE requirement and user fairness on the SE-EE relationship. Analytical and simulation results show that increasing frequency resource is more efficient than increasing spatial resource to improve the SE-EE relationship as a whole. The EE increases with the SE when the frequency resource is not constrained to the maximum value, otherwise a tradeoff between the SE and the EE exists. Sacrificing the fairness among users in terms of ergodic capacities can enhance the SE-EE relationship. In general, the adaptive configuration of spatial and frequency resources outperforms the adaptive configuration of only spatial or frequency resource.

In this paper, we investigate adaptive configuration of spatial and frequency resources to maximize energy efficiency (EE) and reveal the relationship between the EE and the spectral efficiency (SE) in downlink multiple-input-multiple-output (MIMO) orthogonal frequency division multiple access (OFDMA) systems. We formulate the problem as minimizing the total power consumed at the base station under constraints on the average data rates from multiple users, the total number of subcarriers, and the number of radio frequency (RF) chains. We develop a two-step searching algorithm to solve this problem, which first finds the near-optimal numbers of subcarriers for multiple users based on Karush-Kuhn-Tucker (KKT) conditions and then optimize the number of active RF chains. Simulation results demonstrate that increasing frequency resource improves both the SE and the EE, and is more efficient than increasing spatial resource. Consequently, there exists tradeoff between the SE and the EE only when the frequency resource is limited. In general, the adaptive configuration of spatial and frequency resources outperforms the adaptive configuration of only spatial resource and that of only frequency resource.

In this paper, we investigate the configuration of frequency resources from the perspective of maximizing the energy efficiency (EE) of downlink multi-cell multi-carrier multi-antenna systems. We first formulate an optimization problem of subcarrier assignment to minimize the total power consumption at the base stations under the constraints of spectral efficiency (SE) requirements from multiple users. Then we find its closed-form solution by analyzing different cases. Analytical and simulation results show that when the SE requirement is low, using non-overlapped frequency resources is more energy efficient than using overlapped frequency resources and the EE increases with the SE. To support high SE, more spatial resources should be configured but a trade-off between SE and EE appears. Serving cell-center users will provide higher EE, while when serving the cell-edge users maximizing the EE will lead to a minor loss of the SE.

In this paper, power allocation between pilots and data symbols is investigated to maximize energy efficiency (EE) for downlink orthogonal frequency division multiple access (OFDMA) networks. We first derive an EE function considering channel estimation error, which depends on large-scale channel gains of multiple users, allocated power to pilots and data symbols, and circuit power consumption. Then an optimization problem is formulated to maximize the EE under overall transmit power constraint. Exploiting the quasiconcavity property of the EE function, we propose an alternating optimization method in the low transmit power region and reformulate a joint quasiconcave problem in the high transmit power region. Analysis and simulation results show that the power ratio for pilots decreases with the circuit power. When the circuit power is small, the optimal overall transmit power increases with the circuit power. Otherwise, the optimal transmit power does not depend on it. Transmitting more data symbols to the users with higher channel gains improves the EE but at a cost of sacrificing the fairness among multiple users. Simulation results also demonstrate that compared with spectral efficiency (SE)-oriented design, the EE-oriented design can improve the EE performance significantly with a relatively small SE loss.

Fractional frequency reuse (FFR) is an efficient way to mitigate inter-cell interference (ICI) in multi-cell orthogonal frequency division multiple access (OFDMA) networks. In this paper, we investigate the throughput and the optimal threshold for the FFR scheme. The average cell throughputs are derived for both round robin (RR) and maximum SINR (MSINR) scheduling strategies when users are uniformly distributed in the cell region. It is shown from the analysis and simulation results that the throughput increases and the optimal distance threshold decreases with the number of users for both scheduling strategies. The optimal distance threshold approaches the minimum distance that users can be away from the base station when the number of users goes to infinity. The optimal distance threshold increases with the frequency reuse factor of the cell-edge region when the MSINR scheduling is used. The impact of the RR scheduling strategy on the optimal threshold of the FFR scheme is negligible. Simulation also demonstrates that the FFR scheme with the optimal threshold significantly outperforms that with the existing fixed threshold.

In this paper, power allocation between pilots and data symbols is investigated aiming at maximizing energy efficiency(EE) for downlink orthogonal frequency division multiple access (OFDMA) networks. We first derive an EE function when the channel estimation error is considered, which depends on the large-scale channel gains of multiple users, the allocated power to pilots and data symbols, and the circuit power consumption. Then an optimization problem is formulated to maximize the EE under overall transmit power constraint. The relationship between the power for pilots and data symbols is analyzed based on Karush-Kuhn-Tucker (KKT) conditions and the impacts of channel gains on both power allocation and the EE are studied. Exploiting the quasiconcavity property of the EE function, a bisection searching algorithm is developed to find the optimal power allocation. Simulation results demonstrate the performance gain of the proposed optimal power allocation scheme in terms of the EE and the required overall transmit power.

Fractional frequency reuse (FFR) is an efficient method to suppress inter-cell interference (ICI) in multi-cell OFDMA networks. In this paper, the optimal threshold is designed for FFR schemes to identify cell central and edge users. We first derive throughput of the FFR scheme considering small-scale fading channels and different scheduling strategies. Then we conclude from the analysis and simulation results that the throughput of maximum normalized SINR (MNSINR) scheduling is larger than that of round robin scheduling, and the optimal thresholds for both scheduling strategies decrease with the increase of cell user number. Simulation results also show that the performance of FFR scheme with the optimal threshold outperforms that with the conventional predefined threshold.

In this paper, both configuration of active radio frequency (RF) chains and resource allocation are investigated for improving energy efficiency of downlink multiple-input-multiple-output (MIMO) orthogonal frequency division multiple access (OFDMA) systems. We first formulate an optimization problem to minimize the total power consumed at the base station with the maximum transmit power constraint and ergodic capacity constraints from multiple users. Then a two-step suboptimal algorithm is proposed. Specifically, the continuous variable optimization problem is first solved, and then a discretization algorithm is presented to obtain the number of active RF chains and the number of subcarriers allocated to each user. Simulation results demonstrate that the proposed algorithm can provide significant power-saving gain over the all-on RF chain scheme and the adaptive subcarrier allocation helps to save more power.

In a cognitive network where both primary and secondary systems are Orthogonal Frequency Division Multiplexing modulated, carrier frequency offset between the two systems is inevitable and may cause harmful interference. In this paper, we jointly optimize the transceiver for the secondary system considering the frequency offset between secondary transmitter (ST) and primary receiver (PR). We first derive unified interference constraints with different information of inference channels known at ST, and formulate the transceiver design problem based on minimum mean square criterion as a convex optimization problem. To reveal the structure of secondary transceiver, we derive closed-form pre- processors in two special cases. It is shown from the simulations that with the increase of frequency offset, the performance of primary system degrades evidently when its bandwidth is smaller than that of secondary system, whereas the performance of secondary system using the proposed transceiver improves due to frequency diversity.

When both primary and secondary systems are orthogonal frequency division multiplexing modulated and are non-cooperative, carrier frequency offset between the systems is inevitable to cause harmful interference. In this paper, we jointly optimize secondary transceivers assuming that the frequency offset between the secondary transmitter (ST) and the primary receiver (PR) and different channel information from the ST to the PR are known at the ST. We first derive unified interference constraints and obtain the secondary transceivers minimizing the mean square error through convex optimization techniques. We then derive closed-form transceivers for several special cases to reveal the impact of the frequency offset on the secondary transceivers. We show that when there is no frequency offset between the ST and the PR, the optimal processing at the ST is power allocation. Otherwise, both power allocation and precoding are necessary. The impact of the frequency offset on the performance of both systems increases as the interference constraints become tighter and the bandwidth of the primary system becomes smaller. When the proposed transceivers are used, the performance of the secondary system is robust to the frequency offset and the performance of the primary system degrades little due to the remanent frequency offset.