Robotics simulation plays an important role in the design, development, and verification and validation of robotics systems. Simulation represents a potentially cheaper, safer, and more reliable alternative to the widely used practice of manual field testing, and introduces valuable opportunities for extensive test automation. The goal of this paper is to develop a principled understanding of the ways robotics developers use simulation in their testing processes and the challenges they face in doing so. This understanding can guide the improvement of simulators and testing techniques for modern robotics development.To that end, we conduct a survey of 82 robotics developers from a diversity of backgrounds, addressing the current capabilities and limits of simulation in practice. We find that simulation is used by 84% of our participants for testing, and that many participants want to use simulation as part of their test automation. Using qualitative and quantitative research methods, we identify 10 high-level challenges that impede developers from using simulation for manual and automated testing and in general. These challenges include the gap between simulation and reality, a lack of reproducibility, and considerable resource costs associated with simulation. Finally, we outline ways in which simulators can be improved for use as a means of verification and validation and ways that the software engineering community can contribute to these improvements.

We propose a method for detecting execution anomalies in robotics and autonomy software. The algorithm uses system monitoring techniques to obtain profiles of executions. It uses a clustering algorithm to create clusters of those executions, representing nominal execution. A distance metric determines whether additional execution profiles belong to the existing clusters or should be considered anomalies. The method is suitable for identifying faults in robotics and autonomy systems. We evaluate the technique in simulation on two robotics systems, one of which is a real-world industrial system. We find that our technique works well to detect possibly unsafe behavior in autonomous systems.

Decompilation, recovering source code from binary, is useful in many situations where it is necessary to analyze or understand software for which source code is not available. Source code is much easier for humans to read than binary code, and there are many tools available to analyze source code. Existing decompilation techniques often generate source code that is difficult for humans to understand because the generated code often does not use the coding idioms that programmers use. Differences from human-written code also reduce the effectiveness of analysis tools on the decompiled source code. To address the problem of differences between decompiled code and human-written code, we present a novel technique for decompiling binary code snippets using a model based on Recurrent Neural Networks. The model learns properties and patterns that occur in source code and uses them to produce decompilation output. We train and evaluate our technique on snippets of binary machine code compiled from C source code. The general approach we outline in this paper is not language-specific and requires little or no domain knowledge of a language and its properties or how a compiler operates, making the approach easily extensible to new languages and constructs. Furthermore, the technique can be extended and applied in situations to which traditional decompilers are not targeted, such as for decompilation of isolated binary snippets; fast, on-demand decompilation; domain-specific learned decompilation; optimizing for readability of decompilation; and recovering control flow constructs, comments, and variable or function names. We show that the translations produced by this technique are often accurate or close and can provide a useful picture of the snippet's behavior.

As software systems increase in complexity and operate with less human supervision, it becomes more difficult to use traditional techniques to detect when software is not behaving as intended. Furthermore, many systems operating today are nondeterministic and operate in unpredictable environments, making it difficult to even define what constitutes correct behavior. I propose a family of novel techniques to model the behavior of executing programs using low-level signals collected during executions. The models provide a basis for predicting whether an execution of the program or program unit under test represents intended behavior. I have demonstrated success with these techniques for detecting faulty and unexpected behavior on small programs. I propose to extend the work to smaller units of large, complex programs.