Factored Markov Decision Processes (FMDPs) offer a promising framework for overcoming the curse of dimensionality in reinforcement learning (RL) by decomposing high-dimensional MDPs into smaller and independently evolving components. Despite their potential, existing studies on FMDPs face three key limitations: reliance on perfectly factorizable models, suboptimal sample complexity guarantees for model-based algorithms, and the absence of model-free algorithms. To address these challenges, we introduce approximate factorization, which extends FMDPs to handle imperfectly factored models. Moreover, we develop a model-based algorithm and a model-free algorithm (in the form of variance-reduced Q-learning), both achieving the first near-minimax sample complexity guarantees for FMDPs. A key novelty in the design of these two algorithms is the development of a graph-coloring-based optimal synchronous sampling strategy. Numerical simulations based on the wind farm storage control problem corroborate our theoretical findings.

Traditional reinforcement learning (RL) assumes the agents make decisions based on Markov decision processes (MDPs) with one-step transition models. In many real-world applications, such as energy management and stock investment, agents can access multi-step predictions of future states, which provide additional advantages for decision making. However, multi-step predictions are inherently high-dimensional: naively embedding these predictions into an MDP leads to an exponential blow-up in state space and the curse of dimensionality. Moreover, existing RL theory provides few tools to analyze prediction-augmented MDPs, as it typically works on one-step transition kernels and cannot accommodate multi-step predictions with errors or partial action-coverage. We address these challenges with three key innovations: First, we propose the \emph{Bayesian value function} to characterize the optimal prediction-aware policy tractably. Second, we develop a novel \emph{Bellman–Jensen Gap} analysis on the Bayesian value function, which enables characterizing the value of imperfect predictions. Third, we introduce BOLA (Bayesian Offline Learning with Online Adaptation), a two-stage model-based RL algorithm that separates offline Bayesian value learning from lightweight online adaptation to real-time predictions. We prove that BOLA remains sample-efficient even under imperfect predictions. We validate our theory and algorithm on synthetic MDPs and a real-world wind energy storage control problem.

The swift expansion of the ride-hailing industry has given rise to pressing safety issues. This study investigates enhancing safety in the ride-hailing industry by developing an anomaly detection system for detecting abnormal vehicle trajectories. This system defines what constitutes an abnormal trajectory and establishes evaluation metrics for detection methods. Traditional methods for anomaly detection, such as real-time monitoring of speed and direction, are inadequate in the face of data heterogeneity and the sheer volume of information. And the proposed anomaly detection system is designed to overcome these limitations: We introduce a multi-tiered strategy, dividing the anomaly detection task into critical points (including origin and destination) analysis and path analyses to effectively identify and categorize abnormal patterns. By concentrating different features or patterns in each tier, the detection capability of this system is effectively enhanced. Combining our evaluation metrics, the system can provide risk assessment for abnormal critical points and trajectories, thus ensuring the safety of both drivers and passengers.

Color constancy provides stable color features for high-level computer vision tasks such as target recognition and autonomous driving. Existing deep learning-based color constancy algorithms using convolutional neural networks remove the illumination and obtain images under standard illumination. However, these methods suffer from insufficient training data and poor robustness in complex scenes. To address these issues, we propose a new paradigm: self-supervised color constancy with a collaborative generative network (CCCG). CCCG transforms the illumination estimation problem into a generation problem, reducing the solution space and enhancing algorithm robustness in complex scenes. Additionally, CCCG employs a self-supervised network structure, reducing dependence on light source label data. CCCG comprises two network structures: the Filter Network (FN) and the Illumination Network (IN). FN extracts features from the image and generates an image under standard illumination. IN incorporates the extracted physical light source information into the output results of FN and verifies the generated image. Experimental results on the Gehler-Shi and NUS-8 datasets show that CCCG outperforms current color constancy methods across various evaluation metrics and can be applied to other computer vision tasks requiring color constancy preprocessing.

This article explores social manufacturing (SM) within cyber–physical–social systems (CPSSs), leveraging artificial intelligence (AI) to revolutionize energy prosumer networks. We introduce a blockchain-enabled operation and management mechanism for energy systems, incorporating energy aggregators for efficient transaction audits and employing consortium blockchain and proof-of-work for enhanced security. Guided by social governance principles and utilizing the soft actor–critic (SAC) approach for handling renewable generation and load demand uncertainties, our method offers a resilient and cost-effective solution. Simulated case studies reveal a 16.7% reduction in audit costs and a 2.4% increase in peer-to-peer transactions, highlighting improved network synergy. Our approach also reduces redundant trading by 6.5%and cuts operational costs by up to 6%, demonstrating the effectiveness of blockchain in improving cost-efficiency and enhancing social governance and security in energy manufacturing systems. The findings of this study contribute a novel vista to the ongoing discourse in SM, illustrating the formidable potential of advanced information and AI technologies in amplifying the operational acumen of contemporary manufacturing ecosystems.

Fine-grained air pollution data is essential for smart living and efficient city management. However, it is arduous to obtain accurate air pollution data with high spatial and temporal resolutions via mobile crowdsensing (MCS) under limited budgets. Thus, we propose FAD, a system fully using fine-grained air pollution data to provide diverse services. Moreover, a low-cost yet highly accurate portable sensing device is designed for MCS applications to enhance data resolutions. Finally, we demonstrate various FAD-based services for citizens and governments in the real world.

Emergency rescue scenarios are considered to be high-risk scenarios. Using a micro air vehicle (MAV) swarm to explore the environment can provide valuable environmental information. However, due to the absence of localization infrastructure and the limited on-board capabilities, it's challenging for the low-cost MAV swarm to maintain precise localization. In this paper, a collaborative localization system for the low-cost heterogeneous MAV swarm is proposed. This system takes full advantage of advanced MAV to effectively achieve accurate localization of the heterogeneous MAV swarm through collaboration. Subsequently, H-SwarmLoc, a reinforcement learning-based planning method is proposed to plan the advanced MAV with a non-myopic objective in real-time. The experimental results show that the localization performance of our method improves 40% on average compared with baselines.

The utilization of heterogeneous end devices such as the low earth orbit (LEO) satellite, unmanned aerial vehicles (UAVs) and ground users (GUs) deployed at different altitudes, known as the space-air-ground integrated network (SAGI-Net), can be quite promising towards a bunch of advanced applications. Whereas, the energy efficiency of the SAGI-Net communication system is a key criterion needed to be improved urgently in consideration that the inappropriate communication routing will undoubtedly cause a huge communication energy cost of the system especially with a large number of communication devices inside. In this paper, we proposed a novel communication routing selection model for the SAGI-Net system and established a heterogeneous multi-agent reinforcement learning (HMF-MARL) framework to optimize the communication energy efficiency of this system, where the mean-field theory was introduced to enhance the ability of classic MARL method while still maintaining a relatively low computational complexity. The experiment results show that the capacity of the heterogeneous multi-agent system has been improved by nearly 80% using the proposed HMF-MARL method compared with the classic MARL one, which hopefully shows the potential value on the implementation of the SAGI-Net system in the future.

The online ride-hailing platform has revolutionized urban transport. However, there is much room for improvement. We consider meeting the demand for an online ride-hailing at transport hubs late at night, when the public transport system stops its operations. Passengers arriving late at night face a long wait before service. We launch the ride-hailing model in the theoretic framework of queueing and introduce the arrival and the service processes. To improve the efficiency of the ride-hailing platform, as well as to maintain fairness between different types of passengers, we study three variations of carpool service policies. We then provide practical guidelines on the trade-off between efficiency and fairness to assist the online platform designers. Specifically, we derive the analytical trade-off bounds with the passenger parameters. Furthermore, we suggest that these bounds can be good performance estimators for the empirical trade-off when only limited passenger information is available. This analysis motivates us to design the optimal service rate for the entire platform. Finally, we conduct numerical studies based on field data retrieved from Didi Chuxing, highlighting the remarkable performance of our proposed method in terms of improving the quality of online ride-hailing service.

Seismic events can severely damage both electricity and natural gas systems, causing devastating consequences. Ensuring the secure and reliable operation of the integrated energy system (IES) is of high importance to avoid potential damage to the infrastructure and reduce economic losses. This article proposes a new optimal two-stage optimization to enhance the reliability of IES planning and operation against seismic attacks. In the first stage, hardening investment on the IES is conducted, featuring preventive measures for seismic attacks. The second stage minimizes the expected operation cost of emergency response. The random seismic attack is modeled as uncertainty, which is realized after the first stage. An explicit damage assessment model is developed to define the budget set of the uncertain seismic activity. Based on the survivability of transmission lines and gas pipelines of IES, an optimal system investment plan is developed. The problem is formulated as a two-stage distributionally robust optimization (DRO) model, which is tested on an integrated IEEE 30-bus system and 20-node gas network. Case studies demonstrate that the two-stage DRO outperforms robust optimization and a single-stage optimization model in terms of minimizing the investment cost and expected economic loss. This article can help system operators to make economical hardening and operation strategies to improve the reliability of IES under seismic attacks, thus managing a more robust and secure energy system.

This article presents a framework for deploying a minimal number of smart meters to accurately track the ON/OFF states of a massive number of electrical appliances which exploits the sparseness feature of simultaneous ON/OFF switching events of the massive appliances. A theoretical bound on the least number of required smart meters is studied by an entropy-based approach, which qualifies the impact of meter deployment strategies to the state tracking accuracy. It motivates a meter deployment optimization algorithm (MDOP) to minimize the number of meters while satisfying given requirements to state tracking accuracy. To accurately decode the real-time ON/OFF states of appliances by the readings of meters, a fast state decoding (FSD) algorithm based on the hidden Markov model (HMM) is presented to track the state sequence of each appliance for better accuracy. Although traditional HMM needs O ( t 2 2 N ) time complexity to conduct online sequence decoding, FSD improves the complexity to O ( tn U+1 ), where n < N and U is an upper bound of the simultaneous switching events. Both MDOP and FSD are verified extensively using simulations and real PowerNet data. The results show that the meter deployment cost can be saved by more than 80% while still getting over 90% state tracking accuracy.

With the proposed penetration of electric vehicles and advanced metering technology, the demand side is foreseen to play a major role in flexible energy consumption scheduling. On the other hand, the past several years have witnessed utility companies' growing interests to integrate more renewable energy resources. These renewable resources, for example, wind or solar, due to their intermittent nature, brought great uncertainty to the power grid system. In this article, we propose a mechanism that attempts to mitigate the grid operational uncertainty induced by renewable energies by properly exploiting demand flexibility with the help of advanced smart-metering technology. To address the challenge, we develop a novel locational marginal price (LMP)-based pricing scheme that involves active demand-side participation by casting the network objective as a two-stage Stackelberg game between the local grid operator and several aggregators. In contrast to the conventional notion that generation follows load, our game formulation provides more flexibility for the operators and tries to provide adequate incentives for the loads to follow the (stochastic renewable) generation. We use the solution concept of subgame perfect equilibrium to analyze the resulting game. Subsequently, we discuss the optimal real-time conventional capacity planning for the local grid operator to achieve the minimal mismatch between supply and demand with the wind power integration. Finally, we assess our proposed scheme with field data. The simulation results show that our proposed scheme works reasonably well in the long term, even with simple heuristics.