Modern distributed systems face escalating coordination challenges as deployment density scales to thousands of interconnected nodes, from dense wireless networks to large-scale Internet of Things infrastructures. Distributed Constraint Optimization Problems (DCOPs) provide a powerful multi-agent framework that enables distributed decision-making while optimizing global objectives through local interactions, making them particularly well-suited for managing density-driven coordination challenges across various distributed systems. This paper presents a novel DCOP-based group-hierarchical distributed optimization framework addressing these scalability challenges, demonstrated through base station coordination in ultra-dense small cell networks. Our approach separates static pseudotree structures based on agent topology from dynamic constraint values from user equipment measurement reports, eliminating costly tree reconstruction while adapting to changing conditions.We propose a limited solving window algorithm that provides time-bounded optimization and develop a group-based hierarchical approach that constrains coordination dynamics within specific groups and layers, significantly reducing computational overhead while preserving global optimization properties. Extensive experiments in realistic urban environments demonstrate that the proposed approach can significantly improve signal quality by up to 44% compared to traditional uncoordinated networks and 11% compared to centralized optimization with fewer base station parameter adjustments and communication overhead, validating its effectiveness in managing density-driven coordination complexity.

We address the problem of quantum reinforcement learning (QRL) under model-free settings with quantum oracle access to the Markov Decision Process (MDP). This paper introduces a Quantum Natural Policy Gradient (QNPG) algorithm, which replaces the random sampling used in classical Natural Policy Gradient (NPG) estimators with a deterministic gradient estimation approach, enabling seamless integration into quantum systems. While this modification introduces a bounded bias in the estimator, the bias decays exponentially with increasing truncation levels. This paper demonstrates that the proposed QNPG algorithm achieves a sample complexity of $\tilde{\mathcal{O}}(\epsilon^{-1.5})$ for queries to the quantum oracle, significantly improving the classical lower bound of $\tilde{\mathcal{O}}(\epsilon^{-2})$ for queries to the MDP.

This paper investigates the potential of quantum acceleration in addressing infinite horizon Markov Decision Processes (MDPs) to enhance average reward outcomes. We introduce an innovative quantum framework for the agent's engagement with an unknown MDP, extending the conventional interaction paradigm. Our approach involves the design of an optimism-driven tabular Reinforcement Learning algorithm that harnesses quantum signals acquired by the agent through efficient quantum mean estimation techniques. Through thorough theoretical analysis, we demonstrate that the quantum advantage in mean estimation leads to exponential advancements in regret guarantees for infinite horizon Reinforcement Learning. Specifically, the proposed Quantum algorithm achieves a regret bound of $\tilde{\mathcal{O}}(1)$\footnote{$\tilde{\mathcal{O}}(\cdot)$ conceals logarithmic terms of $T$.}, a significant improvement over the $\tilde{\mathcal{O}}(\sqrt{T})$ bound exhibited by classical counterparts, where $T$ is the length of the time horizon.

This paper investigates infinite-horizon average reward Constrained Markov Decision Processes (CMDPs) under general parametrized policies with smooth and bounded policy gradients. We propose a Primal-Dual Natural Actor-Critic algorithm that adeptly manages constraints while ensuring a high convergence rate. In particular, our algorithm achieves global convergence and constraint violation rates of $\tilde{\mathcal{O}}(1/\sqrt{T})$ over a horizon of length $T$ when the mixing time, $\tau_{\mathrm{mix}}$, is known to the learner. In absence of knowledge of $\tau_{\mathrm{mix}}$, the achievable rates change to $\tilde{\mathcal{O}}(1/T^{0.5-\epsilon})$ provided that $T \geq \tilde{\mathcal{O}}\left(\tau_{\mathrm{mix}}^{2/\epsilon}\right)$. Our results match the theoretical lower bound for Markov Decision Processes and establish a new benchmark in the theoretical exploration of average reward CMDPs.

We present the first finite-sample analysis of policy evaluation in robust average-reward Markov Decision Processes (MDPs). Prior work in this setting have established only asymptotic convergence guarantees, leaving open the question of sample complexity. In this work, we address this gap by showing that the robust Bellman operator is a contraction under a carefully constructed semi-norm, and developing a stochastic approximation framework with controlled bias. Our approach builds upon Multi-Level Monte Carlo (MLMC) techniques to estimate the robust Bellman operator efficiently. To overcome the infinite expected sample complexity inherent in standard MLMC, we introduce a truncation mechanism based on a geometric distribution, ensuring a finite expected sample complexity while maintaining a small bias that decays exponentially with the truncation level. Our method achieves the order-optimal sample complexity of $\tilde{\mathcal{O}}(\epsilon^{-2})$ for robust policy evaluation and robust average reward estimation, marking a significant advancement in robust reinforcement learning theory.

As urban populations grow, smart home technology has become a key enabler for enhancing energy efficiency, comfort, and convenience in residential environments. However, existing smart home implementations often struggle to adapt to complex and dynamic user needs due to issues like device heterogeneity, rigid preset solutions, and limited interoperability. This article presents a novel semantic-based self-configuring framework that captures the functional, operational, state, and characteristic perspectives of smart home devices, facilitating seamless interconnection and interoperability across heterogeneous devices. The proposed framework uses service entity, that is semantically modeled to act as a linker and mediator between applications and devices to enable dynamic scheduling and control based on application requests and user preferences. To enable continuous adaptation and learning, the proposed approach formulates the self-configuration problem as an Markov decision process (MDP), with parameters instantiated using semantic information and rules. A novel feedback-based reinforcement learning approach is then proposed to adjust the reward function and state transition parameters based on user feedback. The performances of the proposed approach against a bench-marked fixed and rule-based mechanisms are demonstrated through simulation.

Recent advances in the Internet-of-Things has given rise to the possibility of connecting different kinds of devices to the internet. With low computing resources constraining most IoT devices and their networks, managing the capabilities and services of the various constrained devices over the internet has been an issue for IoT implementations. Existing approaches to this problem lack end-to-end resource-oriented Device Management architecture for resource constrained devices and their networks as a result of protocol fragmentation in the domain. This study first proposes a Device Management architecture which is able to leverage more than one protocol in addressing this problem. We then base on the architecture to put forward an implementation approach to manifesting this solution through our usage of a simulated Smart Building case study. The analysis of the case study shows that the proposed resource-oriented architecture achieves device management and service enablement operations with good performance.

Information sharing is critical to multi-agent team for cooperative decision making in dynamic and partially observable environments. Other than building a full information coverage, if agents in a team can be self-directed to valuate a potential receiver and where to communicate, the coordination efficiency will be greatly enhanced. Although intensive studies of information valuation approaches have been developed in ontology graph matching and natural language processing, these models fail to perform fast reasoning for large-scale decentralized agents dynamic coordination. In this paper, we propose a fast information valuation approach based on a complex network graph model, which helps to indicate the information importance. Similar to vague information valuation by human, the key is that important information always significantly changes their complex information graph with its incorporation. Therefore, we calculate the semantic based value of this new information in a graph model and build a local graph evaluation algorithm to estimate information graph evolution, instead of performing expensive complete graph search. The advantage is that the local valuation algorithm can be easily transformed into efficient queries in agents' information base so that they can make fast decisions. Although the decision may not be precise, similar to human communication, the information sharing performance is good enough to disseminate valuable information in the multi-agent team. We demonstrate the feasibility of the proposed information valuation approach in a multi-agent cooperation case study.

Process management is practical in the state-of-the-art of Internet of things research. However, this has become a bottleneck in recent years since an extreme amount of heterogeneous items have to be recorded and traced with radio frequency identification (RFID) tags. In a typical application of process synthesis management, each item will be involved with multiple processes but when those processes are interconnected, an extremely complex network emerges and has to be managed. Existing works on processing management systems, however, are always case-based and only focus on specific application domains. Thus, the general applied processing management model is rather limited. In this paper, we summarize the characteristics of the RFID application domains, and by abstraction, we propose an innovation of the RFID processing model. In this model, each RFID data were deemed as an operation record and all their processing services are logically interconnected and organized as a procedure graph. In addition, we abstract and summarize the basic RFID data processes into a few types. The advantage of this design is that the basic RFID data processing logic can be preprogrammed and in a real domain, the system can be dynamically programmed by automatically constructing the procedure graph nodes with those basic processes and mapping the interconnection logic according to the topology of the graph. As the last part of this paper, we designed a prototype system for medical instruments infection control to demonstrate our approach.

As intelligence in smart homes increasingly get sophisticated, it has become necessary adopting ontological reasoning in these domains. However, ontologies presently lack a standardised representation for uncertainty in knowledge. A key challenge therefore lies in developing ontology-based decision-making models that can integrate domain uncertainty. In this paper, a decision network extension to OWL ontology using only a subset of domain concepts relevant for probabilistic modelling is proposed. This relevant set of concepts in ontology can be generated on the fly using an algorithm, OWLMB, introduced in this paper. Given an ontology and a class as inputs to OWLMB, Markov boundary of the class is returned as this minimum relevant set. Also, representations for decision and utility nodes are proposed as an extension of valid decision situation to ontology. Validation of this approach in a smart home scenario shows its feasibility in a real application domain.

With the expansion of decentralized multi-robot system's applications, limited communication resources has become a critical factor toward the efficiency of multi-robot coordination when both system sizes scale up and their tasks become more complex. In current researches, it is only possible for every robot in a decentralized setting either to choose a part of communication channels to communicate or interact with the other robots. However, due to the constraint on robots' limited communication capability as well as their local observations of the whole network, each robot's cooperative communication channel choice, based on the partial observation of the states of the system, and communication network is under uncertainty. It is also a major challenge to work out multi-robots' cooperative decision by uncertainty reasoning with only a partial observation to the environment. In this paper, we propose a decentralized approach that allows robots to cooperatively search channels and independently choose channels. The key of this approach is to build an up-to-date observation for each robot's view so that a local decision model is achievable. Based on this model, we simplify the Dec-POMDP problem model and each robot can jointly work out its communication policy in order to improve its local decision utilities for the choice of the communication channel using its current observation of the limited system's states as well as its neighbors' choices. In this way, an approximate balance between timeliness of decision making and cooperatively enhancing the efficiency of using these channels can be reached.

Sharing information is critical to large teamwork for cooperative decision making in dynamic and partially observable environments. To be effective, other than building a full information coverage, agents should valuate how a potential receiver could be benefited with a piece of given information. In this paper, we propose a fast valuation model with complex network based graph modeling and analysis, which help to indicate the information importance to a given information base. Similar to vague information valuation by humans, the key is that important information always significantly changes their complex information graph with its incorporation. Therefore, we calculate the semantic based value of this new information in a graph model and build a local graph evaluation algorithm to estimate information graph evolution, instead of performing expensive complete graph search. Although the decision may be not precise, similar to human communication, it is good enough to disseminate valuable information around the team.