There have been discussions about the value of fidelity since simulation-based training systems have been created. A primary question, which has yet to be fully answered, is what is the effect of level of simulation fidelity on learning on a target task? We present a new analysis method and use it for several analyses of a training simulation for an electronic maintenance task with two levels of fidelity: a high-fidelity simulation that basically takes as much time as the real-world task and a low-fidelity simulation with minimal delays and many actions removed or reduced in fidelity and time. The analyses are based on the Keystroke-Level Model (KLM) and the power law of learning. The analyses predict that the performance on the low-fidelity simulation initially takes between one quarter and one eighth of the time of the high, and thus starts out providing between four and eight times as many practice trials in a given time period. The low-fidelity curve has a lower intercept and a steeper slope. Learners that move from low to high appear to not be adversely affected. For a small number of practice trials, this makes a significant difference. We also explore the effect of missing subtasks in the low-fidelity simulation. This effect varies with the tasks included: If the low-fidelity simulation does not train an important task, learners can be slower when they transfer. We also analyze a simulation that we have built and are studying. These analyses demonstrate that using lower fidelity training situations helps most where there is less time to practice, and that if there is extensive time to practice full fidelity has nearly the same outcome (but perhaps not the same costs or risks). We discuss how this analysis approach can help choose the level of fidelity of future training simulations.

In this paper, we present an experimental study in which an electroencephalogram (EEG) device was used to measure cognitive load in programmers as they attempted to predict the output of C code snippets. Our goal was to see if particular patterns within the snippet induced higher levels of cognitive load, and if the collected EEG data might provide more detailed insights than performance measures. Our results suggest that while cognitive load can be an influence on code comprehension performance, other human factors, such as a tendency to forget certain programming rules or to misread what the code is asking them to do may also play a role, particularly for novice programmers. We conclude that: (1) different types of code patterns can affect programmers' cognitive processes in disparate ways, (2) neither self-reported data nor brainwave activity alone is a reliable indicator of programmers' level of comprehension for all types of code snippets, (3) EEG techniques could be useful to better understand the relationships between program comprehension, code patterns and cognitive processes, and (4) tests like ours could be useful to identify crucial learning gaps in novice programmers, which, in turn can be leveraged to improve programming tools and teaching strategies.

This Research Work in Progress paper presents preliminary data from an experiment study (N = 22) in investigating the effect of learners' motivation on learning outcomes, including learning performance and satisfaction. In STEM fields, learners are required to obtain declarative knowledge as well as apply them to solve real world problems. The flipped strategy has been a popular instructional approach and is used in many classrooms. Although many studies reported positive impact of the flipped strategy, the results are not consistent. The goal of this study is to explore whether learners' motivation in a flipped classroom affect their performance and their satisfaction. We use a quasi-experimental design with two groups based on the initial motivation scores. We find that our participants did not benefit from flipped strategy as much as the traditional strategy. We also found that the participants did not feel better (confident) about the subject. We conclude that students may not be comfortable with the flipped strategy because the flipped strategy is used for only about two weeks and that great instructional attentions may be needed to use the flipped strategy. In the end, we propose areas that are worthy of further investigation.

Program comprehension is a common task in software development. Programmers perform program comprehension at different stages of the software development life cycle. Detecting when a programmer experiences problems or confusion can be difficult. Self-reported data may be useful, but not reliable. More importantly, it is hard to use the self-reported feedback in real time. In this study, we use an inexpensive, non-invasive EEG device to record 8 subjects' brain activity in short program comprehension. Subjects were presented either confusing or non-confusing C/C++ code snippets. Paired sample t-tests are used to compare the average magnitude in alpha and theta frequency bands. The results show that the differences in the average magnitude in both bands are significant comparing confusing and non-confusing questions. We then use ANOVA to detect whether such difference also presented in the same type of questions. We found that there is no significant difference across questions of the same difficulty level. Our outcome, however, shows alpha and theta band powers both increased when subjects are under the heavy cognitive workload. Other research studies reported a negative correlation between (upper) alpha and theta band powers.

Humans often mistake the meaning of source code, and so misjudge a program's true behavior. These mistakes can be caused by extremely small, isolated patterns in code, which can lead to significant runtime errors. These patterns are used in large, popular software projects and even recommended in style guides. To identify code patterns that may confuse programmers we extracted a preliminary set of `atoms of confusion' from known confusing code. We show empirically in an experiment with 73 participants that these code patterns can lead to a significantly increased rate of misunderstanding versus equivalent code without the patterns. We then go on to take larger confusing programs and measure (in an experiment with 43 participants) the impact, in terms of programmer confusion, of removing these confusing patterns. All of our instruments, analysis code, and data are publicly available online for replication, experimentation, and feedback.

We present a work-in-progress project that implements spaced repetition and gamification through mobile application for STEM education as a learning system for K-12 students. Spaced repetition in the classroom has been studied and shown to be effective for foreign language and vocabulary acquisition as well as other types of training, while gamification has similarly been used to improve learners' engagement and motivation. This project combines the advantages of both instructional strategies and delivers it through a mobile learning system by using its ubiquitous nature. We believe by combining spacing and gamification with mobile learning technology, the learning system will yield fruitful results in STEM education. We discuss how existing literature affects our design and our plan of implementing such a system.

Despite the security community's emphasis on the importance of building secure software, the number of new vulnerabilities found in our systems is increasing. In addition, vulnerabilities that have been studied for years are still commonly reported in vulnerability databases. This paper investigates a new hypothesis that software vulnerabilities are blind spots in developer's heuristic-based decision-making processes. Heuristics are simple computational models to solve problems without considering all the information available. They are an adaptive response to our short working memory because they require less cognitive effort. Our hypothesis is that as software vulnerabilities represent corner cases that exercise unusual information flows, they tend to be left out from the repertoire of heuristics used by developers during their programming tasks.

The security community spares no effort in emphasizing security awareness and the importance of building secure software. However, the number of new vulnerabilities found in today's systems is still increasing. Furthermore, old and well-studied vulnerability types such as buffer overflows and SQL injections, are still repeatedly reported in vulnerability databases. Historically, the common response has been to blame the developers for their lack of security education. This paper discusses a new hypothesis to explain this problem by introducing a new security paradigm where software vulnerabilities are viewed as developers' blind spots in their decision making. We argue that such a flawed mental process is heuristic-based, where humans solve problems without considering all the information available, just like taking shortcuts. This paper's thesis is that security thinking tends to be left out by developers during their programming, as vulnerabilities usually exist in corner cases with unusual information flows. Leveraging this paradigm, this paper introduces a novel methodology for capturing and understanding security-related blind spots in Application Programming Interfaces (APIs). Finally, it discusses how this methodology can be applied to the design and implementation of the next generation of automated diagnosis tools.

This study implemented and evaluated an educational computer game as a supplemental instructional method in a fundamental computer programming course. Twenty undergraduates who enrolled in a fundamental computer programming course for non-computer science majors participated in the study. They were randomly assigned to use one of two different supplemental instruction delivery methods for a three-week period of instruction. One group used an electronic quiz program and the other group used an educational computer game. Both methods used the same set of quiz bank and all quizzes were selected randomly by the applications. After the instructional treatments were implemented, students' intrinsic motivation and final achievement scores were measured and analyzed. The result of the study indicated that students in the treatment group had positive attitude toward the educational computer game. Specifically, the students in the educational computer game group demonstrated higher intrinsic motivation and better achievement scores compared to the students in the control group.

Recently, higher education has seen an increasing emphasis on the prominent role of computational thinking in all disciplines. Computational thinking is advocated as not only a fundamental skill or concept in computer science but also a core competency for all disciplines. Teaching students in non-computer science majors computing thinking is challenging because students do not have experts' mental models. This study investigates the knowledge gap that non-computing major college students (n=126) possess about computational thinking in an introductory MS Excel course by measuring their performance using spreadsheet functions in three categories: recall, application, and problem solving. The empirical result, analyzed using ANOVA, shows that students can recall the meaning of those functions but seem to have trouble using them correctly and precisely (cued or uncued). Students' test results suggest the following issues: (1) problems with understanding the data type, (2) failure in translating problems to productive representations using spreadsheet functions, and (3) inadequate stipulation of the computational representations in precise forms. Addressing these problems early and explicitly in future classes could improve the education of computational thinking and alleviate difficulties students may experience in using computational thinking in learning and problem solving.