With the rapid advancements in industrial big data, the Internet of Things, and sensor acquisition technologies, the similarity measurement of multivariate time series has emerged as a pivotal research area in data mining and machine learning. To enhance the accuracy and efficacy of multivariate time series similarity measurement, this paper proposes a sliding window approach based on Transformer. Specifically, each dimension of the multivariate time series is processed through sliding windows and input into a Transformer for feature extraction. By using multiple window sizes, the method simultaneously captures localized temporal segment features and identifies local patterns within the time series. Encoded window features for each sample are combined to form a comprehensive feature sequence that represents the global characteristics of the entire time series. These global features are then used to compute the final similarity measure through Dynamic Time Warping (DTW). This approach effectively captures both local and global features of multivariate time series, significantly improving similarity measurement precision. The effectiveness of the proposed method is validated through 1-Nearest Neighbor (1NN) classification experiments, demonstrating superior accuracy and enhanced performance in similarity measurement. The experiments showed that ten of the sixteen datasets had the best performance in terms of classification accuracy.

Exploration in sparse reward environments remains a significant challenge in reinforcement learning, particularly in Contextual Markov Decision Processes (CMDPs), where environments differ across episodes. Existing episodic intrinsic motivation methods for CMDPs primarily rely on count-based approaches, which are ineffective in large state spaces, or on similarity-based methods that lack appropriate metrics for state comparison. To address these shortcomings, we propose Episodic Novelty Through Temporal Distance (ETD), a novel approach that introduces temporal distance as a robust metric for state similarity and intrinsic reward computation. By employing contrastive learning, ETD accurately estimates temporal distances and derives intrinsic rewards based on the novelty of states within the current episode. Extensive experiments on various benchmark tasks demonstrate that ETD significantly outperforms state-of-the-art methods, highlighting its effectiveness in enhancing exploration in sparse reward CMDPs.

Research in offline reinforcement learning (RL) marks a paradigm shift in RL. However, a critical yet under-investigated aspect of offline RL is determining the subset of the offline dataset, which is used to improve algorithm performance while accelerating algorithm training. Moreover, the size of reduced datasets can uncover the requisite offline data volume essential for addressing analogous challenges. Based on the above considerations, we propose identifying Reduced Datasets for Offline RL (ReDOR) by formulating it as a gradient approximation optimization problem. We prove that the common actor-critic framework in reinforcement learning can be transformed into a submodular objective. This insight enables us to construct a subset by adopting the orthogonal matching pursuit (OMP). Specifically, we have made several critical modifications to OMP to enable successful adaptation with Offline RL algorithms. The experimental results indicate that the data subsets constructed by the ReDOR can significantly improve algorithm performance with low computational complexity.

Comprehending natural language and following human instructions are critical capabilities for intelligent agents. However, the flexibility of linguistic instructions induces substantial ambiguity across language-conditioned tasks, severely degrading algorithmic performance. To address these limitations, we present a novel method named DAIL (Distributional Aligned Learning), featuring two key components: distributional policy and semantic alignment. Specifically, we provide theoretical results that the value distribution estimation mechanism enhances task differentiability. Meanwhile, the semantic alignment module captures the correspondence between trajectories and linguistic instructions. Extensive experimental results on both structured and visual observation benchmarks demonstrate that DAIL effectively resolves instruction ambiguities, achieving superior performance to baseline methods. Our implementation is available at https://github.com/RunpengXie/Distributional-Aligned-Learning.

We present Distributional Soft Actor-Critic (DSAC), a distributional reinforcement learning (RL) algorithm that combines the strengths of distributional information of accumulated rewards and entropy-driven exploration from Soft Actor-Critic (SAC) algorithm. DSAC models the randomness in both action and rewards, surpassing baseline performances on various continuous control tasks. Unlike standard approaches that solely maximize expected rewards, we propose a unified framework for risk-sensitive learning, one that optimizes the risk-related objective while balancing entropy to encourage exploration. Extensive experiments demonstrate DSAC’s effectiveness in enhancing agent performances for both risk-neutral and risk-sensitive control tasks.

In remote sensing image classification, open-set recognition (OSR) poses a significant challenge, aiming to accurately classify known categories while effectively rejecting unknown class samples or identifying potential novel categories. Although existing methods have made strides in recognizing known classes, they exhibit notable limitations in handling unknown class samples. This letter introduces an OSR model for ship detection, termed masked autoencoder (MAE)-based OSR (MOSR), which leverages the robust representation learning capabilities of the MAE. MOSR not only sustains high accuracy in the recognition of known classes but also markedly enhances the performance in the identification of unknown class samples. Comprehensive experiments on the custom RSHIP-137 remote sensing dataset validate the efficacy and superiority of the MOSR model. Compared with the state-of-the-art (SOTA) adversarial reciprocal point learning (ARPL) method, MOSR shows substantial improvements in both known class recognition accuracy and the area under the receiver operating characteristic curve (AUROC) for unknown class recognition for ship detection. This study presents a novel solution for OSR in remote sensing ship detection and offers valuable insights for future research.

Reinforcement learning (RL) has shown strengths in challenging sequential decision-making problems. The reward function in RL is crucial to the learning performance, as it quantifies the degree of task completion. In real-world problems, the rewards are predominantly human-designed, which requires laborious tuning, and is susceptible to human cognitive biases. To achieve automatic auxiliary reward generation, we propose a novel representation learning approach that can measure the “transition distance” between states. Building upon these representations, we introduce an auxiliary reward generation technique for both single-task and skill-chaining scenarios without the need for human knowledge. Furthermore, we theoretically show that the proposed auxiliary rewards maintain the policy invariance property, i.e., the generated rewards will not hurt the policy optimality under the original rewards. In the experiment section, we evaluate the proposed approach in both online and offline learning settings in a wide range of tasks, including robot manipulation and locomotion. The experiment results demonstrate the effectiveness of measuring the transition distance and the induced improvement by auxiliary rewards, which promotes better learning efficiency and increases convergent stability. Beyond that, we demonstrate that the learned manipulation policy with the auxiliary rewards in a simulator can be transferred to the real robot, as shown inhttps://sites.google.com/view/transition-distance-rp/tdrp. Note to Practitioners—The motivation for this paper arises from the need for a technique that enhances robot skill-learning efficiency and performance in both single-task and skill-chaining scenarios. Our research primarily focuses on robot arm manipulation tasks. To accelerate the policy learning process and improve policy performance for executing these tasks, we introduce an auxiliary reward generation technique for both single-task and skill-chaining scenarios without requiring human expertise. This technique leverages the proposed novel representation learning approach, which can measure the “transition distance” between states. During each policy training round, the robot receives a dense reshaped reward created by our approach. Using the policy trained by our method, we successfully control a real Franka Panda robot arm to complete various manipulation tasks.

Many countries are facing energy shortages today and most of the global energy is consumed by HVAC systems in buildings. For the scenarios where the energy system is not sufficiently supplied to HVAC systems, a priority-based allocation scheme based on distributed model predictive control is proposed in this paper, which distributes the energy rationally based on priority order. According to the scenarios, two distributed allocation strategies, i.e., one-to-one priority strategy and multi-to-one priority strategy, are developed in this paper and validated by simulation in a building containing three zones and a building containing 36 rooms, respectively. Both priority-based strategies fully exploit the potential of predictive control solutions. The experiment shows that our scheme has good scalability and achieves the performance of the centralized strategy while making the calculation tractable. Note to Practitioners—The motivation of this paper is to develop a priority-based allocation strategy adapted to energy-limited systems. When energy is limited, the strategy can rationally allocate energy and satisfy the urgent need for energy supply in some specific zones. Two priority strategies are proposed for the case that a single subsystem corresponds to a particular priority and multiple subsystems correspond to the same priority, respectively. The developed strategies have been validated by co-simulation with MATLAB and EnergyPlus in a small-scale three-zone building and a large-scale 36-zone building to show their effectiveness.

Subtask decomposition offers a promising approach for achieving and comprehending complex cooperative behaviors in multiagent systems. Nonetheless, existing methods often depend on intricate high-level strategies, which can hinder interpretability and learning efficiency. To tackle these challenges, we propose a novel approach that specializes subtasks for subgroups by employing diverse observation representation encoders within information bottlenecks. Moreover, to enhance the efficiency of subtask specialization while promoting sophisticated cooperation, we introduce diversity in both optimization and neural network architectures. These advancements enable our method to achieve state-of-the-art performance and offer interpretable subtask factorization across various scenarios in Google Research Football (GRF).

Among the remarkable successes of Reinforcement Learning (RL), self-play algorithms have played a crucial role in solving competitive games. However, current self-play RL methods commonly optimize the agent to maximize the expected win-rates against its current or historical copies, resulting in a limited strategy style and a tendency to get stuck in local optima. To address this limitation, it is important to improve the diversity of policies, allowing the agent to break stalemates and enhance its robustness when facing with different opponents. In this paper, we present a novel perspective to promote diversity by considering that agents could have diverse risk preferences in the face of uncertainty. To achieve this, we introduce a novel reinforcement learning algorithm called Risk-sensitive Proximal Policy Optimization (RPPO), which smoothly interpolates between worst-case and best-case policy learning, enabling policy learning with desired risk preferences. Furthermore, by seamlessly integrating RPPO with population-based self-play, agents in the population optimize dynamic risk-sensitive objectives using experiences gained from playing against diverse opponents. Our empirical results demonstrate that our method achieves comparable or superior performance in competitive games and, importantly, leads to the emergence of diverse behavioral modes. Code is available at https://github.com/Jackory/RPBT.

The large action space is one fundamental obstacle to deploying Reinforcement Learning methods in the real world. The numerous redundant actions will cause the agents to make repeated or invalid attempts, even leading to task failure. Although current algorithms conduct some initial explorations for this issue, they either suffer from rule-based systems or depend on expert demonstrations, which significantly limits their applicability in many real-world settings. In this work, we examine the theoretical analysis of what action can be eliminated in policy optimization and propose a novel redundant action filtering mechanism. Unlike other works, our method constructs the similarity factor by estimating the distance between the state distributions, which requires no prior knowledge. In addition, we combine the modified inverse model to avoid extensive computation in high-dimensional state space. We reveal the underlying structure of action spaces and propose a simple yet efficient redundant action filtering mechanism named No Prior Mask (NPM) based on the above techniques. We show the superior performance of our method by conducting extensive experiments on high-dimensional, pixel-input, and stochastic problems with various action redundancy tasks. Our code is public online at https://github.com/zhongdy15/npm.

Offline reinforcement learning (RL) is crucial for real-world applications where exploration can be costly or unsafe. However, offline learned policies are often suboptimal, and further online fine-tuning is required. In this paper, we tackle the fundamental dilemma of offline-to-online fine-tuning: if the agent remains pessimistic, it may fail to learn a better policy, while if it becomes optimistic directly, performance may suffer from a sudden drop. We show that Bayesian design principles are crucial in solving such a dilemma. Instead of adopting optimistic or pessimistic policies, the agent should act in a way that matches its belief in optimal policies. Such a probability-matching agent can avoid a sudden performance drop while still being guaranteed to find the optimal policy. Based on our theoretical findings, we introduce a novel algorithm that outperforms existing methods on various benchmarks, demonstrating the efficacy of our approach. Overall, the proposed approach provides a new perspective on offline-to-online RL that has the potential to enable more effective learning from offline data.

Pointer meters are widely used in various industries, and there is a growing demand for automatic and non-intrusive access to meter readings. The existing methods of automatically calculating meter readings involve complex processes which are time-consuming and have low automation performance. The paper presents the design of a Separate Dual-Selective Attention Mechanism (SDSAM) module and a Reading Correction Module (RCM). We also introduce the SDSM-DenseNet model, which utilizes dense connections to directly predict meter readings based on meter images. The SDSM-DenseNet model demonstrates high automation performance and reduces reliance on inherent characteristics of the meter. Compared to other attention mechanism models and feature extraction models, the SDSM-DenseNet model achieves lower error rates in calculating meter readings. Furthermore, when compared with representative methods for reading meters, the average error rate of the SDSM-DenseNet model is reduced by approximately 48%, while only requiring 0.021 s to predict a single image.

Energy shortage is a challenge for many countries, and building energy consumption accounts for a considerable proportion of global energy consumption. The main work of this paper is to optimize the energy consumption of heating, ventilating, and air conditioning (HVAC) systems in buildings based on economic model predictive control (EMPC). The cost in EMPC design includes energy consumption and predicted mean vote (PMV), which is an index that evaluates the thermal comfort of indoor occupants. In order to model the nonlinearity of the PMV index, we propose a lattice piecewise linear (PWL) approximation, which has high approximation precision and facilitates the resulting optimization problem, which is basically a piecewise quadratic programming problem. For the piecewise quadratic programming, we propose a descent algorithm that converges quickly and scales well with the length of the prediction horizon in the EMPC problem. The experimental results demonstrate that the proposed method saves 19.78% of the electricity cost compared to the conventional control strategy and significantly increases indoor comfort.Note to Practitioners— The motivation of this article is to provide a control strategy to reduce building energy consumption and ensure indoor thermal comfort. In most of the existing methods for air conditioning temperature control, the occupants’ comfort hasn’t been considered. In this paper, thermal comfort is described by the PMV index, which is basically nonlinear. In order to model the thermal comfort more accurately, in this paper, the PMV index is approximated piecewise linearly in order to meet the requirements of accuracy and computational efficiency. The resulting optimization problem is not hard to solve, and we provide an efficient algorithm for solving this optimization problem. Preliminary simulation experiments demonstrate that this approach is practical, i.e., it achieves energy reduction and ensures thermal comfort. Our strategy, however, has not yet been deployed in real buildings. In future research, we will propose similar techniques for large-scale systems in order to solve energy optimization problems containing multiple thermal zones and realize the proposed technique in real buildings.

Offline reinforcement learning (RL) enables the agent to effectively learn from logged data, which significantly extends the applicability of RL algorithms in real-world scenarios where exploration can be expensive or unsafe. Previous works have shown that extracting primitive skills from the recurring and temporally extended structures in the logged data yields better learning. However, these methods suffer greatly when the primitives have limited representation ability to recover the original policy space, especially in offline settings. In this paper, we give a quantitative characterization of the performance of offline hierarchical learning and highlight the importance of learning lossless primitives. To this end, we propose to use a flow-based structure as the representation for low-level policies. This allows us to represent the behaviors in the dataset faithfully while keeping the expression ability to recover the whole policy space. We show that such lossless primitives can drastically improve the performance of hierarchical policies. The experimental results and extensive ablation studies on the standard D4RL benchmark show that our method has a good representation ability for policies and achieves superior performance in most tasks.

Keeping risk under control is often more crucial than maximizing expected rewards in real-world decision-making situations, such as finance, robotics, autonomous driving, etc. The most natural choice of risk measures is variance, while it penalizes the upside volatility as much as the downside part. Instead, the (downside) semivariance, which captures negative deviation of a random variable under its mean, is more suitable for risk-averse proposes. This paper aims at optimizing the mean-semivariance (MSV) criterion in reinforcement learning w.r.t. steady reward distribution. Since semivariance is time-inconsistent and does not satisfy the standard Bellman equation, the traditional dynamic programming methods are inapplicable to MSV problems directly. To tackle this challenge, we resort to Perturbation Analysis (PA) theory and establish the performance difference formula for MSV. We reveal that the MSV problem can be solved by iteratively solving a sequence of RL problems with a policy-dependent reward function. Further, we propose two on-policy algorithms based on the policy gradient theory and the trust region method. Finally, we conduct diverse experiments from simple bandit problems to continuous control tasks in MuJoCo, which demonstrate the effectiveness of our proposed methods.

Wireless powered edge computing system (WPECS) enhances the computing power and extends the lifetime of wireless devices (WDs). This paper studies the WPECS with multiple WDs, in which the access point (AP) provides some transmission channels which differ from each other in the channel gain, and the WD powered through the wireless power transfer (WPT) technology has some indivisible tasks and adopts binary task-offloading actions. More energy harvested strengthens the WDs with more computing power, while corresponding to more energy consumption. Therefore, how to make the optimal tradeoff between computation rate and energy harvested arises as an interesting issue. To address this issue, this paper first formulates the switch process of transmission channel as a constrained Markov decision process (CMDP), and then proposed an effective algorithm to maximize the sum of computation rates of all WDs in terms of task data bits computed, within the required level of accumulative energy harvested. Theoretical analysis, simulations and field experiments jointly document and illustrate its performance. Note to Practitioners—This paper addresses the interesting tradeoff between computation rate and energy harvested in a wireless powered edge computing system that operates in the environments with limited available energy. It helps to improve the operation efficiency of the edge computing systems in the area of Internet of Things (IoT) or Cyber-Physical Systems (CPS) that employ wireless power transfer technology to power the wireless devices through the access point over the air to maximize the sum of computations rates of all WDs in terms of task data bits computed, while keeping the accumulative energy harvested within a range. Simulations and experimental investigations show that the solution proposed here outperforms existing solutions.

This paper considers the edge computing system (ECS) in which the tasks with dependencies are offloaded and computed in sequence. Different task-offloading orderings come with different ECS memory-cache execution latency (EMCEL) which is caused by writing and reading (WR) the computed results of earlier offloaded tasks between the ECS memory and cache. Therefore, the optimal ordering to offload all the tasks while leading to the minimum EMCEL arises as an interesting issue in practice. This requires to solve a hard exponential explosion optimization problem. To address this issue, this paper first formulates the tasks and their dependencies as a direct acyclic graph (DAG), then converts the exponential explosion problem into a discrete problem that can be solved in polynomial time, and finally develops some theoretical conditions to guide to determine the optimal task-offloading orderings. A novel algorithm called OTOOA to find the optimal task-offloading orderings in polynomial time is proposed. Field experiments show that OTOOA outperforms the existing algorithms. To our best knowledge, this is the initial work towards this issue. Note to Practitioners—For the edge computing system that operates in the application scenarios in which the ECS cache is small while the size of the tasks is relatively large such that it is not allowed for multiple tasks to be processed in the ECS cache parallelly or at the same time, e.g., the execution latency-sensitive and fast big data-processing scenarios in which the multiple tasks depending on the other are offloaded and computed in sequence, this paper helps such edge computing system to improve the operation efficiency with the minimum EMCEL by finding an optimal task-offloading ordering to guide the wireless devices to offload their tasks to the ECS server. Experimental investigations show that the solution proposed here outperforms existing ones.

This paper considers the stopping-time control of multiple smart sensors which independently sample the physical parameter of interests and request the desired data from the common server with probabilities. More samples may lead to higher sensing accuracy but cost more energy. More requests may facilitate the smart sensors to obtain the desired data with higher probability while yet cost more energy. Meanwhile, the heterogeneities of smart sensors issuing the requests may lead to early energy exhaustion for some smart sensors. Therefore, how to make an optimal tradeoff between sensing accuracy and energy consumption while keeping the energy consumption balance arises as an interesting problem. To address this issue, this paper first formulates the stopping-time control policy of an individual smart sensor as a partially observable Markov decision process (POMDP), then coordinates the stopping-time control policies of multiple smart sensors within a noncooperative game (NCG), and finally proposes a noncooperative PODMP game-based algorithm to make the above tradeoff. Theoretical analysis and field experiments jointly document the performance. Note to Practitioners—This paper addresses the interesting tradeoff between sensing accuracy and energy cost while balancing the energy consumption in a sensor network that operates in the environments where the available energy is limited and the early energy exhaustion for some of the smart sensors should be avoided. It helps to improve the operation efficiency of the sensor network with multiple smart sensors and a shared server setting in the area of Internet of Things (IoT) or Cyber-Physical Systems (CPS) to minimize the estimation error, while keeping the accumulative energy cost within a range and balancing the energy consumption. Field experimental investigations show that the solution proposed here outperforms existing solutions.

Offline reinforcement learning (RL) enables effective learning from previously collected data without exploration, which shows great promise in real-world applications when exploration is expensive or even infeasible. The discount factor, $\gamma$, plays a vital role in improving online RL sample efficiency and estimation accuracy, but the role of the discount factor in offline RL is not well explored. This paper examines two distinct effects of $\gamma$ in offline RL with theoretical analysis, namely the regularization effect and the pessimism effect. On the one hand, $\gamma$ is a regulator to trade-off optimality with sample efficiency upon existing offline techniques. On the other hand, lower guidance $\gamma$ can also be seen as a way of pessimism where we optimize the policy’s performance in the worst possible models. We empirically verify the above theoretical observation with tabular MDPs and standard D4RL tasks. The results show that the discount factor plays an essential role in the performance of offline RL algorithms, both under small data regimes upon existing offline methods and in large data regimes without other conservative methods.

Keeping risk under control is often more crucial than maximizing expected reward in real-world decision-making situations, such as finance, robotics, autonomous driving, etc. The most natural choice of risk measures is variance, while it penalizes the upside volatility as much as the downside part. Instead, the (downside) semivariance, which captures the negative deviation of a random variable under its mean, is more suitable for risk-averse proposes. This paper aims at optimizing the mean-semivariance (MSV) criterion in reinforcement learning w.r.t. steady rewards. Since semivariance is time-inconsistent and does not satisfy the standard Bellman equation, the traditional dynamic programming methods are inapplicable to MSV problems directly. To tackle this challenge, we resort to the Perturbation Analysis (PA) theory and establish the performance difference formula for MSV. We reveal that the MSV problem can be solved by iteratively solving a sequence of RL problems with a policy-dependent reward function. Further, we propose two on-policy algorithms based on the policy gradient theory and the trust region method. Finally, we conduct diverse experiments from simple bandit problems to continuous control tasks in MuJoCo, which demonstrate the effectiveness of our proposed methods.

Most of reinforcement learning algorithms optimize the discounted criterion which is beneficial to accelerate the convergence and reduce the variance of estimates. Although the discounted criterion is appropriate for certain tasks such as financial related problems, many engineering problems treat future rewards equally and prefer a long-run average criterion. In this paper, we study the reinforcement learning problem with the long-run average criterion. Firstly, we develop a unified trust region theory with discounted and average criteria. With the average criterion, a novel performance bound within the trust region is derived with the Perturbation Analysis (PA) theory. Secondly, we propose a practical algorithm named Average Policy Optimization (APO), which improves the value estimation with a novel technique named Average Value Constraint. To the best of our knowledge, our work is the first one to study the trust region approach with the average criterion and it complements the framework of reinforcement learning beyond the discounted criterion. Finally, experiments are conducted in the continuous control environment MuJoCo. In most tasks, APO performs better than the discounted PPO, which demonstrates the effectiveness of our approach.

Recently, deep multi-agent reinforcement learning (MARL) has shown the promise to solve complex cooperative tasks. Its success is partly because of parameter sharing among agents. However, such sharing may lead agents to behave similarly and limit their coordination capacity. In this paper, we aim to introduce diversity in both optimization and representation of shared multi-agent reinforcement learning. Specifically, we propose an information-theoretical regularization to maximize the mutual information between agents' identities and their trajectories, encouraging extensive exploration and diverse individualized behaviors. In representation, we incorporate agent-specific modules in the shared neural network architecture, which are regularized by L1-norm to promote learning sharing among agents while keeping necessary diversity. Empirical results show that our method achieves state-of-the-art performance on Google Research Football and super hard StarCraft II micromanagement tasks.

Learning from datasets without interaction with environments (Offline Learning) is an essential step to apply Reinforcement Learning (RL) algorithms in real-world scenarios.However, compared with the single-agent counterpart, offline multi-agent RL introduces more agents with the larger state and action space, which is more challenging but attracts little attention. We demonstrate current offline RL algorithms are ineffective in multi-agent systems due to the accumulated extrapolation error. In this paper, we propose a novel offline RL algorithm, named Implicit Constraint Q-learning (ICQ), which effectively alleviates the extrapolation error by only trusting the state-action pairs given in the dataset for value estimation. Moreover, we extend ICQ to multi-agent tasks by decomposing the joint-policy under the implicit constraint. Experimental results demonstrate that the extrapolation error is successfully controlled within a reasonable range and insensitive to the number of agents. We further show that ICQ achieves the state-of-the-art performance in the challenging multi-agent offline tasks (StarCraft II). Our code is public online at https://github.com/YiqinYang/ICQ.

Web services integrate various components in the Internet of Things (IoT). In a Web service-based data-collection system with multiple smart sensor nodes periodically sampling and estimating the same unknown physical parameter of interest, the smart sensor nodes first submit their estimates to the Web server, and then, the server picking the one with the minimum error seems to be a practical way to arrive at a minimum error estimate (MEE). More submissions provide the Web server with more candidates to consider, which can maximize the probability of the server guaranteeing the MEE, while also leading to more network traffic. Therefore, how to make the optimal tradeoff between network traffic and sensing accuracy arises as an interesting problem. This article proposes a network traffic-dependent probability threshold policy within an intended underlying optimization-theoretical framework to address this problem. The policy is such that the smart sensor nodes submit their estimates and corresponding estimation errors (ECEEs) to the Web server within a tolerable network traffic threshold while maximizing the probability of the server delivering the MEE. Theoretical analysis, simulation, and field experiments document and illustrate its performance.Note to Practitioners—This article addresses the interesting tradeoff between sensing accuracy and network traffic demand in the Web service-based data-collection system that operates in some remote areas with limited network traffic. It helps to improve the operation efficiency of the Internet-of-Things (IoT) systems that employ Web service technology to enable the Web server to deliver minimum error estimate with maximum probability while keeping the network traffic within a given range. Our simulation and experimental investigations show that the solution developed here outperforms existing solutions.

Community detection in complex networks has attracted lots of interest in scientific fields. However, current community detection algorithms mainly focus on unipartite network. In this paper, we propose a new definition of K-partite modularity and a new method which strictly follows the idea of original Louvain algorithm. Compared with other algorithms, our method is more intuitive and easier to implement. We evaluate on both synthetic and real-world networks. Experimental results show that, our method is not only capable to obtain better partitions, but scalable to large-scale data sets.

In Internet of Things systems, Web services enable interoperable machine-to-machine communication over networks. Polling mechanism is a practical way for a Web service-based control system to enable its actuator to respond to its controller under uncertain environments where the actuator does not know exactly when the controller updates its command. Fast response demands a high polling frequency (polling mechanism handles the event in time-driven mode) which may lead to heavy network traffic. Therefore, how to make the optimal tradeoff between the network traffic and delay in a Web service-based control system becomes an interesting problem. This paper formulates the problem of finding the optimal polling frequency control policy of a Web service-based control system as a constrained Markov decision process (CMDP). The policy is such that the actuator responds to the controller within a tolerable delay threshold while minimizing network traffic. An algorithm called CMDPA is proposed to solve the problem. Simulation and field experiments show our policy performance.

Series machines, i.e., machines (which are usually unreliable) arranged in series with no buffering, are pervasive in production systems. In the analysis, design, and optimization of the series-machine system, the efficiency analysis is one of the most fundamental issues. There are not a lot of researches analyzing the efficiency of the series-machine system, and almost all of them assume that the system operates under type-I failure mechanisms (i.e., the breakdown of a machine could make all other series machines forced down) rather than under type-II mechanisms (i.e., the breakdown of a machine does not make any other series machines forced down). The reason that the type-I failure mechanisms are usually assumed in the literature is that the analysis of the series-machine system under type-II mechanisms is much more complex than under type-I mechanisms, although type-II mechanisms are more common in practice. To thoroughly and systematically estimate the efficiency of the series-machine system, in this paper, we propose a unified analytical approach to investigate the efficiency under both type-I and type-II failure mechanisms. Both cases of deterministic and random cycle times are considered. Different from under type-I failure mechanisms, analytical expressions of the efficiency of series-machine systems under type-II failure mechanisms are extremely hard to obtain, and thus, limit bounds of the efficiency are derived and algorithms are developed to calculate its exact value. Results show that the series-machine system under type-II failure mechanisms is more efficient than under type-I mechanisms, which, intuitively making sense, is the reason that type-II mechanisms are more common in the industry.

Cyber security has become a vital issue in cyber-physical systems. This paper analyzes trade-off between security and performance of networked control system, which is related to the time delay by the data encryption and decryption procedure. In addition, we build up an experimental control platform using a remotely controlled servo system, and evaluate the affection of DES encryption system on the dynamical performance of the system.

This paper concerns stopping-time management of smart sensing nodes, a kind of very large scale integration system, based on the tradeoffs between sensing accuracy and power consumption, which are foundations of Internet of Things systems and cyber-physical systems. In practice, smart sensing nodes work periodically, and more samples leads to higher accuracy but more energy cost, so when to stop sampling in a periodic cycle to achieve the optimal tradeoff is an interesting issue. This paper formulates this issue as a partially observable Markov decision process (POMDP) and develops a POMDP-based Optimal Stopping-time Algorithm to make the above tradeoff. Field experiments demonstrate its performance.

For embedded systems with multiple sleep modes, it is interesting to understand how to maximize the energy saving potential by choosing the suitable sleep mode(s) during the idle period. In this paper, we establish a sufficient condition to narrow down the search space of sleep policy and propose a new algorithm: optimal-idle-threshold-policy-algorithm under more realistic setting than the existing works. Theoretical proofs and experimental results justify the benefits of our approach.

Zonal occupant level is of great practical interest for building energy saving under normal operations and for fast evacuation under emergency. Though there are many existing sensing systems to estimate this information, the problem is still challenging due to the privacy concerns, the random human movement, and the accumulative error. In this paper, we consider this important problem and focus on infrared beam systems that monitor the zonal arrival and departure events. We make the following contributions. First, a rule (i.e., Rule 1) based on the stay time is developed to reduce the accumulated estimation error in each zone. Second, a rule (i.e., Rule 2) is designed to coordinate the estimation among neighboring zones. A decentralized estimation method is then developed using these two rules. Third, the advantage of this method is demonstrated through simulation results and field tests. We hope this work brings insight to zonal occupant level estimation in buildings in more general situations.

The Internet of Things (IoT) has received an increasing attention in recent years. Embedded intelligent terminal (EIT), an indispensable part of IoT, works not only as a sensor but also as a primary processor. Due to the limited power resource of EIT, it is important to study how to improve the efficiency of its power use. To tackle this problem, we propose an energy-saving algorithm, Bayesian idle time prediction (BIP). The basic idea of BIP is to explore historical information and obtain a better estimation of idle time. In this paper, we provide a theoretical analysis of BIP and compare our method with three existing algorithms [weighted idle-time-prediction (IP) algorithm, IP algorithm, and running time fixed threshold in IP algorithm] with respect to energy-saving potential, as well as system delay under a random number of tasks. Both simulation and field experiment results demonstrate the advantages of our algorithm in energy saving.

When the output observation noise is output-dependent, identifying the unknown system parameters becomes challenging. Traditional methods based on Mean Square Error, even the ones with corrections still have biased estimations in this case. Many practical cases such as bounded sensor, uncertainty of expression in human involved system identification, and even in physiological or biological model identification actually have this problem. In this paper, some algorithms were proposed to obtain the unbiased estimation of parameters for input-output-nonlinear but identification-linear system under output-dependent bounded noise. We utilized the truncated probability distribution to model the noise and gave the unbiased estimation algorithms of the system parameters as well as noise parameter if unknown. Asymptotic properties of the algorithms indicate that the algorithms converge to the true parameters. Besides illustrative numerical example, we also utilized the algorithm in a real world application to identify the personalized thermal comfort model using human noisy voting data. Results revealed the effectiveness and applicability of the proposed algorithms.

This paper explores energy efficient building design for manufacturing plants. Many efforts have been directed into the field of building design optimization concerning building energy performance, but most of the studies focus on residential buildings or public buildings. Very limited research results studying plants buildings have been reported. However, plants buildings have certain unique features that make the design problem more challenging. Furthermore, the approaches presented in the current publications could not guarantee the performance of their designs if the computation capacity is limited. This paper attempts to address these two issues. First, an EnergyPlus-integrated overall energy consumption estimation framework is developed for a class of manufacturing plants, where the environmental conditions would not affect the energy consumption of the production processes. Based on that, the building design problem for this type of manufacturing plants is formulated as a stochastic programming problem concerning uncertainties arising from the future weather conditions and energy prices, where seasonal production scheduling optimizing is incorporated when estimating the performance of building designs. Second, Ordinal Optimization (OO) method is introduced to solve the problem so as to quantitatively guarantee a high probability of finding satisfactory designs while reducing the computation burden. A numerical example is provided, showing our solution method performs effectively in finding a satisfactory design.

Robustness is an important requirement on many critical networked infrastructures. One of the most important ways to achieve robustness is to keep an appropriate level of redundancy by adding redundant resources for the critical networked infrastructures. Considering wireless with good reconfigurabilities, we add wireless communication capacities to selected nodes of a wired networked system in this paper, where we firstly focus on adding minimum wireless communication capacities to achieve biconnectivity requirements, secondly optimize the wireless capacities installation that maximizes the efficiency of reconfiguration, and thirdly propose an approach of topology reconfiguration with wireless communication to keep system connectivity if there are node failures. To resolve the computational complexity issue in searching for the optimal node combination to add wireless capacities, we propose the concept of the equivalent network that reduces the system topology as a simplified acyclic network. The topology optimization is under a metric of reconfiguration distance which guarantees the maximum of the shortest distances between any one node and that in the set of reconfigurable nodes to be minimal such that the efficiency of topology reconfiguration with node failure is the highest. The performances of the methods presented in the paper are demonstrated through numerical experiments and testing.

The problem of cognitive access of channels of primary users by a secondary user is considered. The transmissions of primary users are modeled as independent continuous-time Markovian on-off processes. A secondary cognitive user employs a slotted transmission format, and it senses one of the possible channels before transmission. The objective of the cognitive user is to maximize its throughput subject to collision constraints imposed by the primary users. The optimal access strategy is in general a solution of a constrained partially observable Markov decision process, which involves a constrained optimization in an infinite dimensional functional space. It is shown in this paper that, when the collision constraints are tight, the optimal access strategy can be implemented by a simple memoryless access policy with periodic channel sensing. Analytical expressions are given for the thresholds on collision probabilities for which memoryless access performs optimally. Extensions to multiple secondary users are also presented. Numerical and theoretical results are presented to validate and extend the analysis for different practical scenarios.

The problem of cognitive access of channels of primary users by a secondary user is considered. The transmissions of primary users are modeled as independent continuous-time Markovian on-off processes. A secondary cognitive user employs slotted transmissions, and it senses one of the possible channels before transmission. The objective of the cognitive user is to maximize its throughput subject to collision constraints imposed by the primary users. The optimal access strategy is in general a solution of a constrained partially observable Markov decision process, which involves a constrained optimization in an infinite dimensional functional space. It is shown in this paper that, when the collision constraints are tight, the optimal access strategy can be implemented by a simple memoryless access policy with periodic channel sensing. Numerical results are presented to validate and extend the analysis for different practical scenarios.

Dynamic spectrum access (DSA) aims to improve spectrum efficiency via spectrum sensing and optimal spectrum access. An essential component in DSA is the joint design of sensing and access strategies. This paper focuses on dynamic spectrum access in the time domain. To maximize channel utilization while limiting interference to primary users, a framework of linear programming is presented based on the stationary distribution of the primary user channels. It is shown that the optimal tradeoff between sensing and transmitting is achieved with required limit on the interference to the primary users.

Group elevator scheduling has received considerable attention due to its importance to transportation efficiency for mid-rise and high-rise buildings. One important trend to improve elevator systems is to collect advance traffic information. Nevertheless, it remains a challenge to develop new scheduling methods which can effectively utilize such information. This paper is to solve the group elevator scheduling problem with advance traffic information. This problem is difficult due to various traffic patterns, complicated car dynamics, and combinatorial explosion of the search space. A two-level formulation is developed with passenger-to-car assignment at the high-level and single car dispatching that is innovatively formulated as passenger-to-trip assignment at the low-level. Detailed car dynamics are embedded in simulation models for performance evaluation. Taking advantage of advance information, a new door action control method is suggested to increase the flexibility of elevators. In view of the hierarchical problem structure, a two-level optimization framework is established. Key problem characteristics are exploited to develop an effective trip-based heuristic for single car dispatching, and a hybrid nested partitions and genetic algorithm method for passenger-to-car assignment which can be extended to solve a generic class of sequential decision problems. Numerical results demonstrate solution quality, computational efficiency, benefit of advance information and the new door action control method, and values of new features in our hybrid method.

In systems of general assembly line with material handling, line-side buffers need to be carefully assigned to a limited number of material delivers (drivers) for part delivery to avoid production stoppage due to material shortage. Such a problem is referred to as line-side buffer assignment problem (LBAP). In this paper, we focus on fixed zoning version of LBAP. We formulate the problem, prove its NP-hardness, and propose an algorithm based on two structural characteristics of the LBAP problem-one being the analogousness between our problem and the parallel machine scheduling (PMS) problem and the other being the monotonicity of the system throughput in the course of assigning line-side buffers to drivers. The developed algorithm globally converges with probability one when there exist feasible assignments. The algorithm is tested on a real system, and the results show that it is effective for solving the LBAP problem.

Performance evaluation of complex manufacturing systems is challenging due to many factors such as system complexity, parameter uncertainties, problem size, just to name a few. In many cases when a system is too complex to model using mathematical formulas, simulation is used as an effective alternative to conduct system analysis. A manufacturing system is a good example of such cases where both system performance and system complexity are greatly impacted by material handling (MH) strategy, management, and operational control. In this paper, we study vehicle general assembly (GA) system with MH, and focus on developing an efficient simulation method for modeling and analysis where traditional simulation methods may suffer from computation intensity. Making use of the partial system decomposability, we introduce an aggregated event-scheduling simulation method with two-level framework. A dividing mechanism with boundary conditions is employed in top-level simulation to divide the global event list into small sizes. A timing-focuses strategy based on max-plus algebra is applied in bottom-level local simulation to further reduce local event lists. With this new method it is possible to mimic real production systems fast and accurately within a reasonable computational time frame. The effectiveness and efficiency of the new simulation method are validated through experimental results.

Maintaining an asset with life-limited parts, e.g., a jet engine or an electric generator, may be costly. Certain costs, e.g., setup cost, can be shared if some parts of the asset are replaced jointly. Reducing the maintenance cost by good joint replacement policies is difficult in view of complicate asset dynamics, large problem sizes and the irregular optimal policy structures. This paper addresses these difficulties by using a rollout optimization framework. Based on a novel application of time-aggregated Markov decision processes, the ldquoOne-Stage Analysisrdquo method is first developed. The policies obtained from the method are investigated and their effectiveness is demonstrated by examples. This method and the existing threshold method are then improved by the ldquorollout algorithmrdquo for the total cost case and the average cost case. Based on ordinal optimization, it is shown that excessive simulations are not necessary for the rollout algorithm. Numerical testing demonstrates that the policies obtained by the rollout algorithms with either the ldquoOne-Stage Analysisrdquo or the threshold method significantly outperform traditional threshold policies.

The maintenance problem with safety-critical components is significant for the economical benefit of companies. Motivated by a practical asset maintenance project, a new joint replacement maintenance problem is introduced in this paper. The dynamics of the problem are modelled as a Markov decision process, whose action space increases exponentially with the number of safety-critical components in the asset. To deal with the curse of dimensionality, we identify a key property of the optimal solution: the optimal performance can always be achieved in a class of policies which satisfy the so-called shortest-remaining-lifetime-first (SRLF) rule. It reduces the action space from 0(2n) to O(n), where n is the number of safety-critical components. To further speed up the optimization procedure, some interesting properties of the optimal policy are derived. Combining the SRLF rule and the neuro-dynamic programming (NDP) methodology, we develop an efficient on-line algorithm to optimize this maintenance problem. This algorithm can handle the difficulties of large state space and large action space. Besides the theoretical proof, the optimality and efficiency of the SRLF rule and the properties of the optimal policy are also illustrated by numerical examples. This work can shed some insights to the maintenance problems in a more general situation.

The problem of dynamically accessing a set of parallel channels occupied by primary users is considered. The secondary user is allowed to sense and to transmit in a single channel. By exploiting idle periods between bursty transmissions of primary users, and by using a periodic sensing strategy, optimal dynamic access is achieved by maximizing the throughput of the secondary user while constraining collision probability with the primary user. The optimal dynamic spectrum access problem can then be formulated within the framework of constrained Markov decision processes (CMDPs). The optimal control policy is identified via a linear program, and its performance is analyzed numerically and through Monte Carlo simulations. Finally, we compare the optimal scheme to an ideal benchmark case when simultaneous sensing of all channels is assumed.

Dynamic voltage scaling is used in energy-limited systems as a means of conserving energy and prolonging their life. We consider a setting in which the tasks performed by such a system are nonpreemptive and aperiodic. Our objective is to control the processing rate over different tasks so as to minimize energy subject to hard real-time processing constraints. Under any given task scheduling policy, we prove that the optimal solution to the offline version of the problem can be efficiently obtained by exploiting the structure of optimal sample paths, leading to a new dynamic voltage scaling algorithm termed the critical task decomposition algorithm (CTDA). The efficiency of the algorithm rests on the existence of a set of critical tasks that decompose the optimal sample path into decoupled segments within which optimal processing times are easily determined. The algorithm is readily extended to an online version of the problem as well. Its worst-case complexity of both offline and online problems is O(N2)

Due to the abundant application background, the optimization of maintenance problem has been extensively studied in the past decades. Besides the well-known difficulty of large state space and large action space, the pervasive application of digital computers forces us to consider the new constraint of limited memory space. The given memory space restricts what strategies can be explored during the optimization procedure. By explicitly quantifying the minimal memory space to store a strategy using support vector machine, we propose to describe simple strategies exactly and only approximate complex strategies. This selective approximation can best utilize the given memory space for any description mechanism. We use numerical results on illustrative examples to show how the selective approximation improves the solution quality. We hope this work sheds some insights to best utilize the memory space for practical engine maintenance strategy optimization problems

Group elevator scheduling is important for transportation efficiency in mid-rise and high-rise buildings, and incessant efforts have been made to improve the service efficiency of elevators. Although these efforts have achieved performance improvements, the performance limit remains an open issue. This paper tries to address that goal by estimating the optimal performance of group elevator scheduling with complete knowledge of future traffic information. A two-level minimization formulation is presented, with passenger-to-car at the high level, and single car dispatching at the low level. The low level is formulated as a passenger-to-trip assignment problem by using a concept trip to facilitate the description of single car dispatching strategies. In view of the difficulty to obtain the absolute optimal performance, our goal turns into its upper and lower bounds. The upper bound is obtained by finding a good feasible solution to this problem. The lower bound is obtained by finding the lower bound for a newly constructed problem whose optimal performance is less than or equal to that of the original problem. Numerical results demonstrate the effectiveness and the scalability of our method

Resource planning for a complex remanufacturing system is in general extremely difficult in terms of, e.g., problem size and uncertainties. In many cases, simulation is the only way to select a good plan among a great number of candidates. When there exist complicated constraints, direct selection could be very inefficient since many candidates may not be feasible but cannot be excluded beforehand. To meet the challenge, a machine learning method is introduced in this paper to perform feasibility analysis. The rough set theory is first applied to establish the relationship between a plan and its feasibility and an iterative reinforcement process is applied to enhance confidence. The numerical testing results show that this method is promising and scalable for the large-scale problems. The research lays a basis for developing an efficient simulation-based optimization method with complicated constraints. Note to Practitioners-This paper was motivated by the resource planning problem for a complex remanufacturing system, which is very important but in general extremely difficult to deal with in terms of, e.g., problem size and uncertainties. Simulation is probably the only way available to select a good plan among a number of candidates. When there exist complicated constraints, simulation becomes even more difficult and selection through simulation could be very inefficient since many candidates may not be feasible but cannot be excluded before simulation. Determining feasibility beforehand is extremely difficult by analytical or numerical methods. This paper suggests a new method using a machine learning-based approach to predict the plan feasibility required in practical applications and can be considered as the first step for optimization-based planning. By applying the rough set theory, the prediction rules are obtained or learned from a training dataset generated by simulation. Then, an iterative reinforcement process is applied to enhance the confidence of learning and to perform iterative retraining on new datasets by the rough set method to generate new rules to add to the knowledge base until the preset threshold is satisfied. The numerical testing results show that the above method is capable of determining the feasible plans for a remanufacturing system with good accuracy. The method is efficient and scalable for the large-scale problems. The method developed in the paper is being incorporated in the framework of ordinal optimization, and a new constrained ordinal optimization method has been developed for remanufacturing planning.

The problem of finding all cycles in the exponentially growing state space of synchronous Boolean networks was studied in the paper by C. Farrow, J. Heidel, J. Maloney, and J. R. Scalar, "Equations for synchronous Boolean networks with biological applications," IEEE trans. Neural Networks, vol. 15, no. 2, pp. 348-354 Mar. 2004. No efficient algorithm was given to solve the problem. We show that even the determination of the number of fixed points (cycles of length 1) for monotone Boolean networks and the determination of the existence of fixed points for general Boolean networks are both strong NP-complete.

A model of multi-part asset with dependent maintenance cost is presented. The problem is to minimize the long-run average cost per time unit. To share some costs, a good policy may jointly replace multiple parts when an asset is maintained. However, it is difficult to obtain an optimal joint replacement policy in view of combinatorial explosion of the states and stochastic system dynamics. To obtain optimal policies for small problems, a novel method is built by recent developed time aggregation Markov decision approach, which leads to analytical and computational simplifications as compared with traditional Markov decision approaches. One-stage and two-stage analysis methods are developed for large problems. The upper bound of one-stage analysis method for single part problems is obtained to show the insight that it can achieve near or true optimal policy. For multi-part problems, they are proved to satisfy certain necessary optimality conditions. These conditions can significantly simplify their implementation. Numerical results show that they are more efficient and effective than other near optimal methods.

Overhaul and repair services are an important segment of the remanufacturing industry, where an asset is disassembled, and component parts are repaired and then assembled to restore the asset to an "as new" condition. A key characteristic of such services is the wide use of rotable parts, i.e., using repaired parts from other assets for assembly, as opposed to waiting for the completion of part repairs from the original assets. However, it is difficult to evaluate the performance of such system. In this paper, and overhaul and repair system characterized by a fork/join structure with rotable inventory is studied. The effects of rotable inventory to system performance are analyzed, and an approximation method is then developed for evaluating key performance measures. By modelling the rotable inventory as a "negative" queue, a system with one overhaul center and one rotable repair shop is first reduced to a tandem queue. Numerical results demonstrate that by this model reduction, TAT can be efficiently estimated within ten percent loss of accuracy as compared with results obtained from hours of simulation. A system with one overhaul center and N rotable repair shops is then reduced to a fork/join system with multiple branches of tandem queues. With appropriate methods to analyze fork/join tandem systems, approximated performance measures can be evaluated.