Yifei Li, Hanane Nour Moussa, Ziru Chen, Shijie Chen, Botao Yu, Mingyi Xue, Benjamin Burns, Tzu-Yao Chiu, Vishal Dey, Zitong Lu, Chen Wei, Qianheng Zhang, Tianyu Zhang, Song Gao, Xuhui Huang, Xia Ning, Nesreen K. Ahmed, Ali Payani, Huan Sun. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

The advancements of language language models (LLMs) have piqued growing interest in developing LLM-based language agents to automate scientific discovery end-to-end, which has sparked both excitement and skepticism about the true capabilities of such agents. In this work, we argue that for an agent to fully automate scientific discovery, it must be able to complete all essential tasks in the workflow. Thus, we call for rigorous assessment of agents on individual tasks in a scientific workflow before making bold claims on end-to-end automation. To this end, we present ScienceAgentBench, a new benchmark for evaluating language agents for data-driven scientific discovery. To ensure the scientific authenticity and real-world relevance of our benchmark, we extract 102 tasks from 44 peer-reviewed publications in four disciplines and engage nine subject matter experts to validate them. We unify the target output for every task to a self-contained Python program file and employ an array of evaluation metrics to examine the generated programs, execution results, and costs. Each task goes through multiple rounds of manual validation by annotators and subject matter experts to ensure its annotation quality and scientific plausibility. We also propose two effective strategies to mitigate data contamination concerns. Using our benchmark, we evaluate five open-weight and proprietary LLMs, each with three frameworks: direct prompting, OpenHands, and self-debug. Given three attempts for each task, the best-performing agent can only solve 32.4% of the tasks independently and 34.3% with expert-provided knowledge. These results underscore the limited capacities of current language agents in generating code for data-driven discovery, let alone end-to-end automation for scientific research.

In this paper, we consider an energy sustainable wireless Internet of Things (IoTs) network with a reconfigurable intelligent surface (RIS). Specifically, the Hybrid Access Point (HAP) performs beamforming to transfer energy to a set of devices, and the devices then use the harvested energy for Non-orthogonal multiple access (NOMA)-based data transmissions, where a RIS is employed to enhance both the energy harvesting and data transmissions. An optimization problem is formulated to maximize the sum-rate of IoT devices by jointly optimizing the energy beamforming of the HAP, the phase shifts of RIS, the selection of devices in NOMA transmissions with power control, and the time allocation for energy harvesting. As the formulated optimization problem is a complex mixed-integer non-linear programming (MNLP) problem, we decompose it into four subproblems and apply Block Coordinate Descent (BCD) to iteratively optimize each subproblem until convergence is achieved. A novel joint optimization algorithm is proposed to select a subset of devices with transmission power control to attain the maximum sum-rate. A closed-form expression for the optimal time allocation is further derived to strike a balance between the energy harvesting and the data transmissions, considering the residual energy resulting from the previous power control. Simulations validate that the proposed solution outperforms state-of-the-art algorithms in the literature.

In this paper, we examine how large language models (LLMs) solve multi-step problems under a language agent framework with three components: a generator, a discriminator, and a planning method.We investigate the practical utility of two advanced planning methods, iterative correction and tree search.We present a comprehensive analysis of how discrimination accuracy affects the overall performance of agents when using these two methods or a simpler method, re-ranking.Experiments on two tasks, text-to-SQL parsing and mathematical reasoning, show that: (1) advanced planning methods demand discriminators with at least 90% accuracy to achieve significant improvements over re-ranking; (2) current LLMs' discrimination abilities have not met the needs of advanced planning methods to achieve such improvements; (3) with LLM-based discriminators, advanced planning methods may not adequately balance accuracy and efficiency.For example, compared to the other two methods, tree search is at least 10-20 times slower but leads to negligible performance gains, which hinders its real-world applications.1

Diverse critical data, such as location information and driving patterns, can be collected by IoT devices in vehicular networks to improve driving experiences and road safety. However, drivers are often reluctant to share their data due to privacy concerns. The Federated Vehicular Network (FVN) is a promising technology that tackles these concerns by transmitting model parameters instead of raw data, thereby protecting the privacy of drivers. Nevertheless, the performance of Federated Learning (FL) in a vehicular network depends on the joining ratio, which is restricted by the limited available wireless resources. To address these challenges, this paper proposes to apply Non-Orthogonal Multiple Access (NOMA) to improve the joining ratio in a FVN. Specifically, a vehicle selection and transmission power control algorithm is developed to exploit the power domain differences in the received signal to ensure the maximum number of vehicles capable of joining the FVN. Our simulation results demonstrate that the proposed NOMA-based strategy increases the joining ratio and significantly enhances the performance of the FVN. Index Terms—Federated Vehicular Network, NOMA

Adopting serverless computing to edge networks benefits end-users from the pay-as-you-use billing model and flexible scaling of applications. This paradigm extends the boundaries of edge computing and remarkably improves the quality of services. However, due to the heterogeneous nature of computing and bandwidth resources in edge networks, it is challenging to dynamically allocate different resources while adapting to the burstiness and high concurrency in serverless workloads. This article focuses on serverless function provisioning in edge networks to optimize end-to-end latency, where the challenge lies in jointly allocating wireless bandwidth and computing resources among heterogeneous computing nodes. To address this challenge, We devised a context-aware learning framework that adaptively orchestrates a wide spectrum of resources and jointly considers them to avoid resource fragmentation. Extensive simulation results justified that the proposed algorithm reduces over 95% of converge time while the end-to-end delay is comparable to the state of the art.

With tremendous efforts on developing effective e-commerce models, conventional e-commerce models show limited success in generalist e-commerce modeling, and suffer from unsatisfactory performance on new users and new products – a typical out-of-domain generalization challenge. Meanwhile, large language models (LLMs) demonstrate outstanding performance in generalist modeling and out-of-domain generalizability in many fields. Toward fully unleashing their power for e-commerce, in this paper, we construct ECInstruct, the first open-sourced, large-scale, and high-quality benchmark instruction dataset for e-commerce. Leveraging ECInstruct, we develop eCeLLM, a series of e-commerce LLMs, by instruction-tuning general-purpose LLMs. Our comprehensive experiments and evaluation demonstrate that eCeLLM models substantially outperform baseline models, including the most advanced GPT-4, and the state-of-the-art task-specific models in in-domain evaluation. Moreover, eCeLLM exhibits excellent generalizability to out-of-domain settings, including unseen products and unseen instructions, highlighting its superiority as a generalist e-commerce model. Both the ECInstruct dataset and the eCeLLM models show great potential in empowering versatile and effective LLMs for e-commerce. ECInstruct and eCeLLM models are publicly accessible through this link.

In this paper, we propose a novel beamforming codebook (CB) design for wireless communications with recon-figurable intelligent surface (RIS) in both near-field and far-field scenarios. To this end, we first develop a generic model to analyze spherical waves and explore the boundary between the near-field and far-field regions. Based on this model, we propose a novel beamforming design for both the near-field and far-field areas. We prove that near-field beamforming can be mathematically decomposed into two parts: directional beamforming which is similar to far-field beamforming, and distance beamforming within the near field region. In addition, we propose an algorithm to decide the beamforming CB, taking into account the practical constraint of quantized phase shifts in RIS implementation. Finally, we implement the proposed beamforming CB design in a network scenario, considering both cases with and without the location information of the user equipment (UE). Extensive simulations validate the superior performance of the proposed beamforming design.

In-context learning with large language models (LLMs) has recently caught increasing attention due to its superior few-shot performance on various tasks.However, its performance on text-to-SQL parsing still has much room for improvement.In this paper, we hypothesize that a crucial aspect of LLMs to improve for text-to-SQL parsing is their multi-step reasoning ability.Thus, we systematically study how to enhance LLMs' reasoning ability through chain of thought (CoT) style prompting, including the original chain-of-thought prompting (Wei et al., 2022b) and least-to-most prompting (Zhou et al., 2023).Our experiments demonstrate that iterative prompting as in Zhou et al. (2023) may be unnecessary for text-to-SQL parsing, and using detailed reasoning steps tends to have more error propagation issues.Based on these findings, we propose a new CoT-style prompting method for text-to-SQL parsing.It brings 5.2 and 6.5 point absolute gains on the Spider development set and the Spider Realistic set, respectively, compared to the standard prompting method without reasoning steps; 2.4 and 1.5 point absolute gains, compared to the least-to-most prompting method 1 .

Lingbo Mo, Shijie Chen, Ziru Chen, Xiang Deng, Ashley Lewis, Sunit Singh, Samuel Stevens, Chang-You Tai, Zhen Wang, Xiang Yue, Tianshu Zhang, Yu Su, Huan Sun. Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue. 2023.

As a main use case of 5G and Beyond wireless network, the ever-increasing machine type communications (MTC) devices pose critical challenges over MTC network in recent years. It is imperative to support massive MTC devices with limited resources. To this end, Non-orthogonal multiple access (NOMA) based random access network has been deemed as a prospective candidate for MTC network. In this paper, we propose a deep reinforcement learning (RL) based approach for NOMA-based random access network with truncated channel inversion power control. Specifically, each MTC device randomly selects a pre-defined power level with a certain probability for data transmission. Devices are using channel inversion power control yet subject to the upper bound of the transmission power. Due to the stochastic feature of the channel fading and the limited transmission power, devices with different achievable power levels have been categorized as different types of devices. In order to achieve high throughput with considering the fairness between all devices, two objective functions are formulated. One is to maximize the minimum long-term expected throughput of all MTC devices, the other is to maximize the geometric mean of the long-term expected throughput for all MTC devices. A Policy based deep reinforcement learning approach is further applied to tune the transmission probabilities of each device to solve the formulated optimization problems. Extensive simulations are conducted to show the merits of our proposed approach.

In this paper, we analytically study the performance of non-orthogonal multiple access (NOMA) transmissions in a random access network with truncated channel inversion power control. Specifically, in a slotted ALOHA network in support of NOMA transmissions, a wireless device randomly selects the transmission power with a certain probability, using channel inversion power control yet subject to the upper bound of the transmission power. Taking into consideration the stochastic nature of wireless fading channels, we first quantify two network areas such that devices in different areas have various choices of transmission powers for NOMA transmissions. An analytical model is developed to analyze the successful transmission probability and throughput of wireless devices located in different areas. Based on the analysis, two optimization problems are formulated to maximize the network throughput and the minimum throughput of wireless devices by tuning the transmission probabilities of each device. To solve the formulated combinatorial optimization problems, two heuristic algorithms are proposed. Extensive simulations are conducted to validate the analysis, and verify the efficiency of the proposed algorithm to attain the maximum network throughput and max-min fairness.

It is known that when the constellation points of the combined signal are close to each other in conventional SIC, the received signals will be corrupted severely by mutual interference. To tackle this problem, an improved Successive Interference Cancellation (SIC) in the uplink Non-orthogonal multiple access (NOMA) system has been proposed in this paper. Specifically, artificial time offsets have been inserted into each transmission to achieve higher degrees of freedom for the power levels of users and cancel the mutual interference. Besides, inserted time offsets can introduce additional resources to increase the accuracy of recognizing the superimposed signals at the receiver side. Our goal is to minimize the error probability of the users' transmission by searching for optimal power allocation. Computer simulation under the 3-user case displays that the improved SIC's bit error rate (BER) performance outperforms the P-NOMA with other SIC-based schemes.

In this article, a cybertwin-assisted nonorthogonal random access (RA) system is presented, where the cybertwins of the physical devices at the access point (AP) collect the devices’ information and decide the transmission parameters on behalf of the devices to achieve the maximum system performance. Specifically, the system performance of a$p$-persistent slotted CSMA system with nonorthogonal multiple access (NOMA) is analyzed, in which wireless devices transmit data to the ensure the received signal strength at the AP side is either high power or low power with certain probabilities. We first develop an analytical framework to quantify the successful transmission probability and the sum data rate as a function of the above probabilities. Accordingly, the feasible region of the number of high-power and low-power devices to ensure successful transmission is derived. With the analysis, nonconvex optimization problems are then formulated to maximize successful transmission probability and the sum data rate, respectively. To tackle the nonconvexity, an effective and fast-convergent iterative algorithm is designed to obtain the optimal transmission probabilities for the devices. Extensive simulations are conducted to validate our analytical results and demonstrate the benefits of NOMA in RA networks.

Channel aggregation (CA) is proposed in IEEE 802.11ax to allow wireless users to aggregate multiple available channels, either contiguous or non-contiguous, to improve the network throughput. In this paper, the performance of CA is extensively investigated. It is shown that a simple CA that aggregates all available channels does not always promote but may degrade the network performance due to the increased inter-channel contentions in a random access wireless local area network (WLAN). Thus, it is of critical importance to select an appropriate set of channels for CA. To this end, we propose an efficient probabilistic channel aggregation scheme to maximize the network throughput under the quality of service constraints. That is, an ax user aggregates each secondary channel with a certain probability based on the traffic load of the secondary channel. A Proximal Policy Optimization (PPO) based approach is further applied to intelligently tune the aggregating probabilities of secondary channels to maximize the network throughput. Numerical results show that the proposed algorithm can greatly improve the network throughput compared with existing CA algorithms in the literature.

Non-orthogonal multiple access (NOMA) has been considered as a promising solution for improving the spectrum efficiency of next-generation wireless networks. In this paper, the performance of a p-persistent slotted ALOHA system in support of NOMA transmissions is investigated. Specifically, wireless users can choose to use high or low power for data transmissions with certain probabilities. To achieve the maximum network throughput, an analytical framework is developed to analyze the successful transmission probability of NOMA and long term average throughput of users involved in the non-orthogonal transmissions. The feasible region of the maximum number of concurrent users using high and low power to ensure successful NOMA transmissions are quantified. Based on analysis, an algorithm is proposed to find the optimal transmission probabilities for users to choose high and low power to achieve the maximum system throughput. In addition, the impact of power settings on the network performance is further investigated. Simulations are conducted to validate the analysis.

Internet of Things (IoT) systems in general consist of a lot of devices with massive connectivity. Those devices are usually constrained with limited energy supply and can only operate at low power and low rate. One solution to limited energy is to use energy harvesting to provide sustainable energy. The set of technologies adopted in next-generation wireless communication systems, such as massive MIMO and Non-Orthogonal Multiple Access (NOMA), can provide solutions to increase the throughput of IoT systems. In this paper we investigate a cellular-based IoT system combined with energy harvesting and NOMA. We consider all base stations (BS) and IoT devices follow the Poisson Point Process (PPP) distribution in a given area. The unit time slot is divided into two phases, energy harvesting phase in downlink (DL) and data transmission phase in uplink (UL). That is, IoT devices will first harvest energy from all BS transmissions and then use the harvested energy to do the NOMA information transmission. We define an energy harvesting circle within which all IoT devices can harvest enough energy for NOMA transmission. The design objective is to maximize the total throughput in UL within the circle by varying the duration T of energy harvesting phase. In our work, we also consider the inter-cell interference in the throughput calculation. The analysis of Probability Mass Function (PMF) for IoT devices in the energy harvesting circle is also compared with simulation results. It is shown that the BS density needs to be carefully set so that the IoT devices in the energy harvesting circle receive relatively smaller interference and energy circles overlap only with small probability. Our simulations show that there exists an optimal T to achieve maximum throughput. When the BSs are densely deployed consequently the total throughput will decrease because of the interference.

In this paper, we consider a green cloud radio access network (C-RAN) with simultaneous wireless and power transfer ability. In order to reduce the energy consumed for updating the channel state information (CSI), energy users are divided into two different groups, including the free charge group and the MIMO group. Then a semi-definite programming problem is formulated under the constraints of energy and information transmission requirements. To minimize the total energy consumption, two algorithms are developed to authorize the energy users into two group divisions in single time slot. Then the algorithms are extended to long term scenarios consisting training and long term stages, the CSI of free charge energy users are not required during the long term stage. Simulation and numerical results are presented to demonstrate the efficiency of the proposed algorithms in significantly reducing the energy consumption of C-RAN systems.

In this paper, we investigate joint beamforming for energy-throughput tradeoff in a sustainable cloud radio access network system, where multiple base stations (BSs) powered by independent renewable energy sources will collaboratively transmit wireless information and energy to the data receiver and the energy receiver simultaneously. In order to obtain the optimal joint beamforming design over a finite time horizon, we formulate an optimization problem to maximize the throughput of the data receiver while guaranteeing sufficient RF charged energy of the energy receiver. Although such problem is non-convex, it can be relaxed into a convex form and upper bounded by the optimal value of the relaxed problem. We further prove tightness of the upper bound by showing the optimal solution to the relaxed problem is rank one. Motivated by the optimal solution, an efficient online algorithm is also proposed for practical implementation. Finally, extensive simulations are performed to verify the superiority of the proposed joint beamforming strategy to other beamforming designs.