Facilitating efficient collaboration among multiple robots in complex smart environments presents significant challenges. Multi-agent reinforcement learning (MARL) offers a viable solution for training robots in task selection, navigation, and collaboration. However, classical MARL methods that rely solely on local observations result in the loss of global awareness, leading to suboptimal performance. To tackle this, many collaborative MARL methods employ centralized learning to utilize complete information during training. However, acquiring such global information in real-world multi-robot systems is often impractical and computationally expensive, necessitating an abstraction method to enhance decision-making and accelerate the learning process. In this research, we propose a novel semi-centralized MARL (SC-MARL) framework that enhances communication among robots through a shared knowledge base called the Awareness Map (AM). We introduce three variations of the Awareness Map, Uniform-AM, Linear-AM, and Sigmoid-AM, and evaluate their performance in a predefined multi-robot environment. Using Unity3D, we designed various smart environments with different complexities, incorporating diverse learning objectives and cooperative challenges. Experimental results show that our AM-enabled SC-MARL effectively trains robot groups, achieving 50% higher performance than decentralized learning methods. Additionally, the adaptability of our approach supports transfer learning, allowing robots to reuse knowledge across multiple scenarios and learn faster than starting from scratch.

Multi-Agent Reinforcement Learning (MARL) has emerged as a powerful paradigm for solving complex tasks involving both cooperation and competition among agents. However, the decision-making processes of MARL models often difficult to interpret, presenting challenges for transparency and user trust. While existing explainable reinforcement learning techniques predominantly focus on single-agent settings, they fall short in addressing the intricacies of multi-agent environments. To bridge this gap, we propose PEIM (Policy Explanation in MARL), a temporally aware framework for explaining decision-making in MARL using local and model-agnostic techniques. PEIM captures and visualizes recent decision histories of agents, highlighting influential features and actions to provide both real-time and temporal insights. These explanations support temporal pattern mining, strategic behavior analysis, and behavior modeling, thereby facilitating a deeper understanding of emergent multiagent dynamics. We validate the proposed framework through experiments on competitive multi-agent scenarios in the StarCraft Multi-Agent Challenge using pre-trained MARL models. The results show that PEIM enabled the identification of individual behavior patterns (e.g., fleeing, target selection) and collaborative strategies (e.g., focus fire, grouping). These interpretable insights align agent decisions with human reasoning, effectively reducing the black-box nature of MARL systems and fostering trust through enhanced transparency and interpretability.

As cloud computing becomes foundational to sectors like healthcare, finance, and artificial intelligence, accurate resource utilization forecasting has emerged as a critical challenge for ensuring efficiency, cost-effectiveness, and service reliability. This research introduces Runtime-Assembled Context-Specific Cooperative Artificial Neural Networks (RACANN), a novel modular neural framework designed to predict cloud resource utilization with improved interpretability and efficiency. Unlike monolithic deep learning models, RACANN dynamically assembles lightweight context-specific neural agents at runtime, enabling fine-grained temporal adaptability and significantly reducing computational overhead. We further propose eRACANN, an explainable extension that embeds a fuzzy logic-based linguistic layer, offering human-readable justifications for predictions. Experimental results across both structured and noisy real-world datasets demonstrate that while deep neural networks (DNNs) may achieve marginally lower error rates, eRACANN excels in modularity, interpretability, and contextual transparency which are critical properties for operational deployment in dynamic, mission-critical cloud environments. The RACANN framework offers a scalable path toward explainable, adaptive, and resource-efficient AI for cloud infrastructure management.

Information sharing among agents to jointly solve problems is challenging for multi-agent reinforcement learning algorithms (MARL) in smart environments. In this paper, we present a novel information sharing approach for MARL, which introduces a Team Information Matrix (TIM) that integrates scenario-independent spatial and environmental information combined with the agent's local observations, augmenting both individual agent's performance and global awareness during the MARL learning. To evaluate this approach, we conducted experiments on three multi-agent scenarios of varying difficulty levels implemented in Unity ML-Agents Toolkit. Experimental results show that the agents utilizing our TIM-Shared variation outperformed those using decentralized MARL and achieved comparable performance to agents employing centralized MARL.

This innovative practice full paper describes a novel twofold approach to analyze the data collected during a six-week research-based summer professional development workshop for middle and high school STEM teachers in Southwest Missouri. In the fast-evolving field of Computer Science (CS), particularly within the Internet of Things (IoT) domain, the demand for a skilled workforce is increasing. To meet this demand, it is crucial to provide K-12 students with education in computing, computational thinking, and other broader Science, Technology, Engineering, and Mathematics (STEM) disciplines. The existing STEM curriculum could be enhanced to develop skills such as research-based problem-solving more effectively, aiming for a more comprehensive skill set among students. Therefore, empowering STEM teachers with a solid foundation of research-based problem-solving skills can significantly boost their readiness for the classroom, thus enriching their students' educational experiences. The Research Opportunity for Smart Environments (ROSE) program, a three-year initiative funded by the National Science Foundation (NSF), aims to prepare middle and high school STEM teachers to effectively introduce CS concepts through innovative IoT applications in Smart Environments, including Smart Homes and Smart Classrooms. To support STEM education in rural and other underrepresented areas, a cohort of in-service teachers from rural Southwest Missouri was selected for the inaugural ROSE summer workshop. Our data collection approach included open-ended questions and effective formative assessment techniques to gather the teachers' perspectives during the initial summer session. We adopted a twofold data analysis strategy, using the qualitative coding tool MAXQDA and sentiment analysis tools Valence Aware Dictionary and sEntiment Reasoner (VADER) and TextBlob, to analyze the teachers' responses. The research questions explored the impact of the ROSE program on educators and how their experiences and outlooks influenced their engagement and professional development within the program. The findings indicate that the ROSE program has positively influenced the teachers' personal and professional growth. The teachers experienced increased confidence and knowledge in research and teaching CS concepts, despite facing various challenges, which acted as motivation for learning. Their reflections also indicated that the mentorship and resources provided by the ROSE program have promoted the development of innovative teaching methods and a shift towards a student-centered approach, showcasing the program's success in fostering the personal and professional development of teachers for the advancement of future CS and STEM professionals.

With the rise of the Internet of Things (IoT), smart homes can provide intelligent services to monitor household appliances remotely and automate user tasks. However, a significant amount of human intervention is expected in the deployment and operation of such services, making it inconvenient for human users who are less tech-savvy. Therefore, such systems should be trained to learn user behavior patterns to automatically configure and adapt their actions according to the preferences and daily routines of the occupants with minimal user involvement to enhance user experience. This paper uses an unsupervised learning approach that can be integrated with a generative policy framework. It enables automatic operational policy generation by analyzing the continuous data from sensors and smart devices. In order to generate policies according to user preferences, our process infers users' behavior patterns by looking for the patterns in their daily routine activities. We compared the performance of our proposed learning approach with existing approaches used in generative policy frameworks. Evaluation results show that our proposed approach positively contributes to the automatic policy generation to automate user tasks in smart homes.

Multi-Agent Reinforcement Learning (MARL) algorithms are widely adopted in tackling complex tasks that require collaboration and competition among agents in dynamic Multi-Agent Systems (MAS). However, learning such tasks from scratch is arduous and may not always be feasible, particularly for MASs with a large number of interactive agents due to the extensive sample complexity. Therefore, reusing knowledge gained from past experiences or other agents could efficiently accelerate the learning process and upscale MARL algorithms. In this study, we introduce a novel framework that enables transfer learning for MARL through unifying various state spaces into fixed-size inputs that allow one unified deep-learning policy viable in different scenarios within a MAS. We evaluated our approach in a range of scenarios within the StarCraft Multi-Agent Challenge (SMAC) environment, and the findings show significant enhancements in multi-agent learning performance using maneuvering skills learned from other scenarios compared to agents learning from scratch. Furthermore, we adopted Curriculum Transfer Learning (CTL), enabling our deep learning policy to progressively acquire knowledge and skills across pre-designed homogeneous learning scenarios organized by difficulty levels. This process promotes inter- and intra-agent knowledge transfer, leading to high multi-agent learning performance in more complicated heterogeneous scenarios.

Intelligent connected vehicle is rapidly growing with the 5-G technology; the diversity of functional interfaces has significantly expanded the avenues of attack, making automotive controller area network (CAN) more vulnerable to cyberthreats. Automotive CAN network attacks are a direct threat to traffic safety, and in this study, we explore the use of intrusion detection techniques for mitigating cyberattacks. However, most automotive CAN network intrusion detection technologies are not capable of defending against sophisticated attacks, making it extremely challenging for detecting intrusions in practice. In this article, we propose a novel time interval conditional entropy method for detecting intrusions in automotive CAN networks. The time interval conditional entropy intrusion detection method is not susceptible to interference and is capable of detecting a variety of attacks. In our experiments, the conditional entropy values of regular communication messages are collected and utilized to distinguish and detect the attacks. The time interval conditional entropy detection method is implemented and evaluated in our controller area net-work bus (CAN-BUS) network platform. The experiments show that our method has higher detection accuracy and is easier to deploy compared to existing automotive CAN network intrusion detection methods.

Distributed decision-making in multi-agent systems presents difficult challenges for interactive behavior learning in both cooperative and competitive systems. To mitigate this complexity, MAIDRL presents a semi-centralized Dense Reinforcement Learning algorithm enhanced by agent influence maps (AIMs), for learning effective multi-agent control on StarCraft Multi-Agent Challenge (SMAC) scenarios. In this paper, we extend the DenseNet in MAIDRL and introduce semi-centralized Multi-Agent Dense-CNN Reinforcement Learning, MAIDCRL, by incorporating convolutional layers into the deep model architecture, and evaluate the performance on both homogeneous and heterogeneous scenarios. The results show that the CNN-enabled MAIDCRL significantly improved the learning performance and achieved a faster learning rate compared to the existing MAIDRL, especially on more complicated heterogeneous SMAC scenarios. We further investigate the stability and robustness of our model. The statistics reflect that our model not only achieves higher winning rate in all the given scenarios but also boosts the agent’s learning process in fine-grained decision-making.

In recent years, reinforcement learning algorithms have been used in the field of multi-agent systems to help the agents with interactions and cooperation on a variety of tasks. Controlling multiple agents simultaneously is extremely challenging as the complexity increases drastically with the number of agents in the system. In this study, we propose a novel semi-centralized deep reinforcement learning algorithm, MAIDRL, for mixed cooperative and competitive multi-agent environments. Specifically, we design a robust DenseNet-style actor-critic structured deep neural network for controlling multiple agents based on the combination of local observation and abstracted global information to compete with opponent agents. We extract common knowledge through influence maps considering both enemy and friendly agents for unit positioning and decision-making in combat. Compared to the centralized method, our design promotes a thorough understanding of the potential influence that a unit has without the need for a complete view of the global state. In addition, this design enables multiagent understanding of common goals, unlike fully decentralized methods. The proposed method has been evaluated on StarCraft Multi-Agent Challenge scenarios in the real-time strategy game, StarCraft II, and the results show that, statistically, the agents controlled by MAIDRL perform better than or as well as those controlled by centralized and decentralized methods.

In this paper, we propose a new hybrid structure learning method that incorporates case-injected genetic algorithms as a score-and-search method for determining Bayesian network structure from data. In our approach, we first find the probabilistic dependencies among variables to constrain the search space and then employ case-injected genetic algorithms in the score-and-search phase to find a quality structure from the reduced search space. The novelty of our work lies with the introduction of combining case-based reasoning with genetic algorithms to evolve a near-optimal Bayesian network in fewer generations compared to a randomly initialized genetic algorithm. Our case-injected genetic algorithms enhance Bayesian network structure learning performance over a sequence of similar problems by extracting and storing knowledge from previously solved problems and utilizing the accumulated knowledge to solve subsequent similar problems. To evaluate the viability of our proposed approach, we conducted a series of experiments by generating a sequence of similar problems based on using data sets obtained randomly from three well-known benchmark Bayesian networks. We also compared the performance of our proposed approach with the state-of-the-art algorithm. Our preliminary results show that case-injected genetic algorithms provide better performance in learning Bayesian network structure compared to GA and the state-of-the-art algorithm. Our proposed approach has applications in real-world domains such as e-commerce system and health care.

We investigate a multi-objective, cooperative, co-evolutionary algorithm to evolve control tactics for groups composed from multiple types of units in real-time strategy games. Such tactical control or micromanagement of units is called micro. Building on prior work, we cooperatively co-evolve micro for a ranged unit using a parameterized control algorithm along with micro for a melee unit using a pure potential fields approach and show that we can simultaneously co-evolve micro for melee and ranged units. These cooperatively co-evolved control algorithms for melee and ranged units evolve to work well together to defeat the default Starcraft II AI, even when outnumbered. We are also able to generate manually co-evolved AI that is significantly better than the default Starcraft II AI and defeat it to generate human competitive micro for controlling multiple types of units. Furthermore, using a multi-objective fitness function leads to a pareto front of near-optimal micro behaviors that range from fleeing while sustaining minimal damage to fighting and maximizing damage to opponents. Such a pareto front naturally provides a user or AI player a variety of micro behaviors suitable for the different types of situations encountered in real-time strategy games. We believe these results indicate the potential of our method for generating effective micro for multiple types of units in real-time strategy games with application in multi-agent control, robotics, and other heterogeneous system control problems.

We compare three promising approaches to micromanaging units in real-time strategy games. These approaches span the range from easily understandable meta-search, which uses genetic algorithms to search through the space of parameters of a human specified control algorithm to pure potential fields, which searches through a space of less human understandable potential field parameter values, to neuro-evolution of augmented topologies which evolves an opaque difficult to understand neural network. All three approaches use a two-objective pareto optimal fitness function that maximizes damage done to opponent units and minimizes damage received by friendly units. We first show that all three approaches can quickly evolve micro superior to the default AI for Starcraft 2, a popular real-time strategy game and research testbed. We then manually co-evolve micro against previously evolved micro to produce micro that plays well against good (gold level) human Starcraft 2 players. Furthermore, we can integrate the micro produced by different approaches to control a single group of units composed from multiple types. These results indicate that we may choose our approach based on our need to understand unit control behavior and thus provides another bridge to transferring research results from autonomous units in simulation games to autonomous agents (robots) in the real world.

Learning the structure of a Bayesian network from data is complex because the number of possible structures increases super-exponentially with the increase in the number of nodes. To address this problem, we propose a hybrid approach comprised of two phases: the constraint-based phase that identifies dependencies among variables to minimize the search space, followed by a score-and-search phase which employs a genetic algorithm to evolve the Bayesian network from the reduced search space. We evaluate the performance of our approach by comparing it with existing algorithms on a limited amount of data sets generated from three benchmark networks. The results illustrate that the proposed algorithm achieves good performance in learning the structure particularly for medium to large networks. Next, we apply our method to a new data set generated from a hand-designed network - the RoRSS (Rules of the Road Ship Simulator). The preliminary results indicate that our method is also satisfactory for small networks with a limited amount of data. The work presented here is a proof-of-concept for our proposed approach aimed at discovering knowledge from data samples of varying sizes and in the presence of small to high number of nodes. Based on the results obtained so far, we are confident that our method is suitable to efficiently learn the structure of the RoRSS network from a large data set.

We attack the problem of controlling teams of autonomous units during skirmishes in real-time strategy games. Earlier work had shown promise in evolving control algorithm parameters that lead to high performance team behaviors similar to those favored by good human players in real-time strategy games like Starcraft. This algorithm specifically encoded parameterized kiting and fleeing behaviors and the genetic algorithm evolved these parameter values. In this paper we investigate using influence maps and potential fields alone to compactly represent and control real-time team behavior for entities that can maneuver in three dimensions. A two-objective fitness function that maximizes damage done and minimizes damage taken guides our multi-objective evolutionary algorithm. Preliminary results indicate that evolving friend and enemy unit potential field parameters for distance, weapon characteristics, and entity health suffice to produce complex, high performing, three-dimensional, team tactics.

We attack the problem of evolving high performance micro behaviors in 3D RTS-game-like simulations. Prior work had shown the potential for the Meta-Search approach to evolve high performance micro for RTS games like StarCraft. We extend this work by moving to 3D and by moving to more realistic physics for simulating the movement of entities in our RTS-game-like simulation. We compare the evolved micro performance of our entities with different physics models of motion on the same scenarios against identical opponent units in a 3D RTS simulation. Results show that our genetic algorithm approach works to reliably evolve high quality 3D micro behaviors for entities independent of the physics model used. Furthermore, experiments show that the entity's acceleration has more of an effect on performance than rotation speed. Our work provides evidence for the generalizability of an evolutionary approach to generating complex behavior for 3D RTS games, training simulations, and real-world unmanned vehicles.

We attack the problem of predicting nitrate concentrations in a stream by using a genetic algorithm to minimize the difference between observed and predicted concentrations on hydrologic nitrate concentration model based on a US Geological Survey collected data set. Nitrate plays a significant role in maintaining ecological balance in aquatic ecosystems and any advances in nitrate prediction accuracy will improve our understanding of the non-linear interplay between the factors that impact aquatic ecosystem health. We compare the genetic algorithm tuned model against the LOADEST estimation tool in current use by hydrologists, and against a random forest, generalized linear regression, decision tree, and gradient booted tree and show that the genetic algorithm does statistically significantly better. These results indicate that genetic algorithms are a viable approach to tuning such non-linear, hydrologic models.

We are interested in using genetic algorithms to generate winning maneuvering behaviors (or micro) in skirmish scenarios for three dimensional Real-Time Strategy games. In prior work, we encoded parameterized 3D micro behaviors like target selection and kiting into an algorithm for controlling friendly units in battle. Genetic algorithms then tuned these parameters to guide unit maneuvering in order to win skirmishes. In this study, we investigate a new representation for micro behaviors that uses only an influence map and a combination of thirteen potential fields. Genetic algorithms then tune influence map and potential field parameters to evolve winning micro behaviors. We compare the performance of both representations on identical scenarios against identical opponents in a full 3D RTS game environment called FastEcslent. The results show that the genetic algorithm using our new representation using less domain knowledge, reliably evolved high quality 3D micro behaviors that slightly, but significantly, outperformed behaviors from our prior work. Our work thus provides evidence for the viability of using potential fields for generating high quality, complex, micro for three dimensional RTS games.

We investigate and develop a coevolutionary approach to finding strong, robust build orders for real-time strategy games. Which units to produce and the order in which to produce them is one important aspect of real-time strategy gameplay. In real-time strategy games, creating plans to address unit production problems are called “build orders.” Our research compares build orders produced from a coevolutionary algorithm, genetic algorithm (GA), and hill climber (HC) to exhaustive search. GAs find the strongest build orders, while coevolution produces more robust build orders than a genetic algorithm or HC. Case injection into the coevolutionary teachset and population can be used to bias coevolution into producing build orders that beat specific opponents or play like specific players, while maintaining robustness. Finally, in this paper, we extend our representation by adding branching and iteration to the build-action sequence and show that this more complex representation enables coevolution to find stronger build orders. We believe this study is a start toward a promising approach for creating strong, robust build orders for RTS games.

We investigate heuristic search algorithms to generate high-quality micromanagement in combat scenarios for real-time strategy (RTS) games. Macro- and micromanagement are two key aspects of RTS games. While good macro helps a player collect more resources and build more units, good micro helps a player win skirmishes and battles against equal numbers and types of opponent units or win even when outnumbered. In this paper, we use influence maps and potential fields as a basis representation to evolve short-term positioning and movement tactics. Unit microbehaviors in combat are compactly encoded into 14 parameters. A genetic algorithm evolves good microbehaviors by manipulating these 14 parameters. We compared the performance of our evolved ECSLBot with two other state-of-the-art bots, UAlbertaBot and Nova, on several skirmish scenarios in a popular RTS game StarCraft. The results show that the ECSLBot tuned by genetic algorithms outperforms UAlbertaBot and Nova in kiting efficiency, target selection, and fleeing. Further experiments show that the parameter values evolved in one scenario work well in other scenarios and that we can switch between preevolved parameter sets to perform well in unseen scenarios containing more than one type of opponent unit. We believe our representation and approach applied to each unit type of interest can result in effective microperformance against melee and ranged opponents and provides a viable approach toward complete RTS bots.

This paper investigates the use of Genetic Algorithms (GAs) to evolve defensive strategies against iterated and memory enabled induction attacks in cognitive radio networks. Security problems in cognitive radio networks have been heavily studied in recent years. However, few studies have considered the effect of memory size on attack and defense strategies. We model cognitive radio network attack and defense as a zero-sum stochastic game. Our research focuses on using GAs to recognize attack patterns from different attackers and evolving defensive strategies against the attack patterns so as to maximize network utility. We assume attackers are not only able to attack high utility channels, but are also capable of attacking based on the history of high utility channel usage by the secondary user. In our simulations, different memory lengths are used by the secondary user against memory enabled attackers. Results show that the best performance strategies evolved by GAs gain more payoff, on average, than the Nash equilibrium. Against our baseline memory enabled attackers, GAs quickly and reliably found the theoretically globally optimal defensive strategy. These results indicate that GAs is a viable approach for generating strong defenses against arbitrary memory based attackers.

We compare genetic algorithms against hill-climbers for generating competitive unit micro-management for winning real-time strategy game skirmishes. Good group positioning and movement, which are part of unit micro-management can help win skirmishes against equal numbers and types of opponent units or even when outnumbered. In this paper, we use influence maps to generate group positioning and potential fields to guide unit movement. We tested the behaviors obtained from genetic algorithm and two types of hill-climbing search against the default Starcraft AI using the brood war API. Preliminary results show that while our hill-climbers quickly find influence maps and potential fields that generate quality positioning and movement in our simulations, they only find quality solutions fifty to seventy percent of the time. On the other hand, genetic algorithms evolve high quality solutions a hundred percent of the time, but take significantly longer.