With the use of Deep Learning (DL) in safety-critical domains, the systematic testing of these systems has become a critical issue for human life. Due to the data-driven nature of Deep Neural Networks (DNNs), the effectiveness of tests is closely related to the adequacy of test datasets. Test data need to be labeled, which requires manual human effort and sometimes expert knowledge. DL system testers aim to select the test data that will be most helpful in identifying the weaknesses of the DNN model by using resources efficiently. To help achieve this goal, we propose a test data prioritization approach based on using a meta-model that gets uncertainty metrics as input, which are derived from outputs of other base models. Integrating different uncertainty metrics helps overcome individual limitations of these metrics and be effective in a wider range of scenarios. We train the meta-models with the objective of predicting whether a test input will lead the tested model to make an incorrect prediction or not. We conducted an experimental evaluation with popular image classification datasets and DNN models to evaluate the proposed approach. The results of the experiments demonstrate that our approach effectively prioritizes the test datasets and outperforms existing state-of-the-art test prioritization methods used in comparison. In the experiments, we evaluated the test prioritization approach from a distribution-aware perspective by generating test datasets with and without out-of-distribution data.

Playtesting is an essential step in the game design process. Game designers use the feedback from playtests to refine their designs. Game designers may employprocedural personasto automate the playtesting process. In this article, we present two approaches to improve automated playtesting. First, we proposedeveloping persona, which allows a persona to progress to different goals. In contrast, the procedural persona is fixed to a single goal. Second, a human playtester knows which paths she has tested before, and during the consequent tests, she may test different paths. However, reinforcement learning (RL) agents disregard these previous paths. We propose a novel methodology that we refer to as alternative path finder (APF). We train APF with previous paths and employ APF during the training of an RL agent. APF modulates the reward structure of the environment, while preserving the agent’s goal. When evaluated, the agent generates a different trajectory that achieves the same goal. We use the general video game artificial intelligence and VizDoom frameworks to test our proposed methodologies. We use proximal policy optimization RL agent during experiments. First, we compare the playtest data generated by developing and procedural persona. Our experiments show that developing persona provides better insight into the game and how different players would play. Second, we present the alternative paths found using APF and argue why traditional RL agents cannot learn those paths.

In this article, we present a new methodology that employs tester agents to automate video game testing. We introduce two types of agents - synthetic and humanlike - and two distinct approaches to create them. Our agents are derived from Sarsa and Monte Carlo tree search (MCTS) but focus on finding defects, while traditional game-playing agents focus on maximizing game scores. The synthetic agent uses test goals generated from game scenarios, and these goals are further modified to examine the effects of unintended game transitions. The humanlike agent uses test goals extracted by our proposed multiple greedy-policy inverse reinforcement learning (MGP-IRL) algorithm from tester trajectories. MGP-IRL captures multiple policies executed by human testers. We use our agents to produce test sequences, and run the game with these sequences. At each run, we use an automated test oracle to check for bugs. We analyze the proposed method in two parts-we compare the success of humanlike and synthetic agents in bug finding, and we evaluate the similarity between humanlike agents and human testers. We collected 427 trajectories from human testers using the General Video Game Artificial Intelligence (GVG-AI) framework and created three games with 12 levels that contain 45 bugs. Our experiments reveal that humanlike and synthetic agents compete with human testers' bug finding performances. Moreover, we show that MGP-IRL increases the humanlikeness of agents while improving the bug finding performance.

In this paper, we study the effects of several Monte Carlo Tree Search (MCTS) modifications for video game testing. Although MCTS modifications are highly studied in game playing, their impacts on finding bugs are blank. We focused on bug finding in our previous study where we introduced synthetic and human-like test goals and we used these test goals in Sarsa and MCTS agents to find bugs. In this study, we extend the MCTS agent with several modifications for game testing purposes. Furthermore, we present a novel tree reuse strategy. We experiment with these modifications by testing them on three testbed games, four levels each, that contain 45 bugs in total. We use the General Video Game Artificial Intelligence (GVG-AI) framework to create the testbed games and collect 427 human tester trajectories using the GVG-AI framework. We analyze the proposed modifications in three parts: we evaluate their effects on bug finding performances of agents, we measure their success under two different computational budgets, and we assess their effects on human-likeness of the human-like agent. Our results show that MCTS modifications improve the bug finding performance of the agents.

Virtual reality (VR) provides a completely digital world of interaction which enables the users to modify, edit, and transform digital elements in a responsive way. Mixed reality (MR), which is the result of blending the digital world and the physical world together, brings new advancements and challenges to human, computer and environment interactions. This paper focuses on adapting the already-existing methods and tools in architecture to both VR and MR environments under sustainable architectural design domain. For this purpose, we benefit from the semantically enriched data platforms of Building information modelling (BIM) tools, the performance calculation functions of building energy simulation tools while transcending these data into VR and MR environments. In this way, we were able to merge these diverse data for the virtual design activity. Nine participants have already tested the initial prototype of MR-based only interaction environment in our previous study [1]. According to the feedbacks, the user interface and interaction mechanisms were updated and the environment was made accessible also in VR. These updates made four types of interactions possible in MR and VR: 1) MR environment using HoloLens with gestures, 2) MR environment using HoloLens with a clicker, 3) VR environment using HTC Vive with two controllers, and 4) HoloLens emulator with a mouse. All these interaction cases were tested by 21 architecture students in an in-house workshop. In this workshop, we collected data on presence, usability, and technology acceptance of these cases. Our results show that interaction in a VR environment is the most natural interaction type and the participants were eager to use both MR and VR environments instead of an emulator. To our best of knowledge, this is the first comparative study of a BIM-based architectural design medium in both VR and MR environments.