WEKA is a widely used, open-source machine learning platform. Due to its intuitive interface, it is particularly popular with novice users. However, such users often find it hard to identify the best approach for their particular dataset among the many available. We describe the new version of Auto-WEKA, a system designed to help such users by automatically searching through the joint space of WEKA's learning algorithms and their respective hyperparameter settings to maximize performance, using a state-of-the-art Bayesian optimization method. Our new package is tightly integrated with WEKA, making it just as accessible to end users as any other learning algorithm.

We describe Mechanical TA, an automated peer review system, and report on our experience using it over three years. Mechanical TA differs from many other peer review systems by involving human teaching assistants (TAs) as a way to assure review quality. Human TAs both evaluate the peer reviews of students who have not yet demonstrated reviewing proficiency and spot check the reviews of students who have. Mechanical TA also features "calibration" reviews, allowing students to quickly gain experience with the peer-review process. We used Mechanical TA for weekly essay assignments in a class of about 70 students, a course design that would have been impossible if every assignment had had to be graded by a TA. We show evidence that it helped to support student learning, leading us to believe that the system may also be useful to others.

Modern surveillance systems for practical applications with diverse and mobile sensors are large, complex, and expensive. It is known that unexpected behaviors can emerge from such systems, and when these behaviors correspond to weaknesses in a surveillance system, we call them emergent vulnerabilities. Given their cost and importance to security, it is essential to test these systems for such vulnerabilities prior to deployment. To that end, we automate the testing process with multiagent systems and machine learning. However, the conventional—and most intuitive–approach is to focus the machine learning on the subject system, which leads to a high‐dimensional problem that is intractable. Instead, we demonstrate in this paper that learning attacks on the system is tractable and provides a viable testing method. We demonstrate this with a series of studies in simulation and with a small‐scale model system featuring elements typically found in real physical surveillance systems. Our machine learning method finds successful attacks in simulation, which we can duplicate with the physical system. The method is scalable, with the implication that it could be used to test larger, real surveillance installations.

Many different machine learning algorithms exist; taking into account each algorithm's hyperparameters, there is a staggeringly large number of possible alternatives overall. We consider the problem of simultaneously selecting a learning algorithm and setting its hyperparameters, going beyond previous work that attacks these issues separately. We show that this problem can be addressed by a fully automated approach, leveraging recent innovations in Bayesian optimization. Specifically, we consider a wide range of feature selection techniques (combining 3 search and 8 evaluator methods) and all classification approaches implemented in WEKA's standard distribution, spanning 2 ensemble methods, 10 meta-methods, 27 base classifiers, and hyperparameter settings for each classifier. On each of 21 popular datasets from the UCI repository, the KDD Cup 09, variants of the MNIST dataset and CIFAR-10, we show classification performance often much better than using standard selection and hyperparameter optimization methods. We hope that our approach will help non-expert users to more effectively identify machine learning algorithms and hyperparameter settings appropriate to their applications, and hence to achieve improved performance.

Large, complex systems can exhibit unforeseen behaviours. In the case of surveillance and security systems, these behaviours can be weaknesses that should be discovered by automated testing and ameliorated. Previous work has shown that such automated testing can be done using particle swarm optimization to learn behaviours that allow a set of attackers to defeat the system. However, for the optimization to succeed, it must have some knowledge about what constitutes a successful attack in order to guide the swarm. This knowledge is encapsulated in a goal ordering structure. In this paper, we examine the goal ordering structure and its role in the learning of system weakness. We specifically look at applications in harbour surveillance and security, and show how knowledge of the likely properties of a successful attack can be added to the goal ordering structure. Our experimental results show that adding knowledge to the goal ordering structure improves the search, when that knowledge is correctly inserted into the structure.

We present a general scheme for testing multiagent systems, respectively policies used by them, for unwanted emergent behavior using learning of cooperative behavior via particle swarm systems. By using particle swarm systems in this setting, we are able to create agents interacting/attacking the tested agents that can use parameterised high-level actions. We also can evaluate the quality of an attack using several measures that can be prioritised and used in a multi-objective manner in the search. This solves some general problems of other testing approaches using learning. We instantiate this general scheme to test harbour patrol and interception policies for two Canadian harbours, showing that our approach is able to find problems in these policies.

In robot building, attention tends to focus on internal signal processing and the way desired motor signals are generated. But equally important is the selection and configuration of the robot's sensory resources. The degree to which sensory input informs the process of output generation directly impacts internal complexity since the more informative input signals are, the less internal processing is required. In the extreme case, input signals may identify motor signals exactly and internal processing may then be eliminated completely. The paper presents an information theoretic model for measuring this tradeoff being input information and internal processing complexity, and establishes the conditions under which it is feasible to utilize the processing-free strategy of ‘direct connection’. The model is then used for purposes of analysing five robots, including two commercial products (Sony AIBO and WowWee's Robosapiens). The principled exploitation of very informative sensory input is then shown to be effective in an unexpectedly wide range of situations.