Sequential learning involves learning tasks in a sequence and proves challenging for most neural networks. Biological neural networks regularly succeed at the sequential learning challenge and are even capable of transferring knowledge both forward and backward between tasks. Artificial neural networks often totally fail to transfer performance between tasks and regularly suffer from degraded performance or catastrophic forgetting on previous tasks. Models of associative memory have been used to investigate the discrepancy between biological and artificial neural networks due to their biological ties and inspirations, of which the Hopfield network is the most studied model. The dense associative memory (DAM), or modern Hopfield network, generalizes the Hopfield network, allowing for greater capacities and prototype learning behaviors while still retaining the associative memory structure. We give a substantial review of the sequential learning space with particular respect to the Hopfield network and associative memories. We present the first published benchmarks of sequential learning in the DAM using various sequential learning techniques and analyze the results of the sequential learning to demonstrate previously unseen transitions in the behavior of the DAM. This letter also discusses the departure from biological plausibility that may affect the utility of the DAM as a tool for studying biological neural networks. We present our findings, including the effectiveness of a range of state-of-the-art sequential learning methods when applied to the DAM, and use these methods to further the understanding of DAM properties and behaviors.

Abstract Information present in any training set of vectors for machine learning can be interpreted in two different ways, either as whole states or as individual atomic units. In this paper, we show that these alternative information distributions are often inherently incongruent within the training set. When learning with a Boltzmann machine, modifications in the network architecture can select one type of distributional information over the other; favouring the activation of either state exemplar or atomic characteristics. This choice of distributional information is of relevance when considering the representation of knowledge in logic. Traditional logic only utilises preference that is the correlate of whole state exemplar frequency. We propose that knowledge representation derived from atomic characteristic activation frequencies is the correlate of compositional typicality, which currently has limited formal definition or application in logic. Further, we argue by counter-example, that any representation of typicality by ‘most preferred model semantics’ is inadequate. We provide a definition of typicality derived from the probability of characteristic features; based on neural network modelling.

Abstract The field of belief revision in logic is still in evolution and holds a variety of disparate approaches; a consequence of theoretical conjecture. As a probabilistic model of supra-classical, non-monotonic (SCNM) logic, the Boltzmann machine, offers an experimental gateway into the field. How does the Boltzmann network adapt to new information? Catastrophic forgetting is the default response to retraining in any neural network. We have moderated this irrational non-monotonicity by alterations in the Boltzmann learning algorithm. The spectrum of experimental belief change is limited by the availability of ‘new’ information, a pragmatic realization co-related to the property of Rational Monotonicity in the domain of SCNM logic. Recognizing this upper boundary of defeasible belief simplifies the task of experimentally exploring machine adaptation. A minority of belief revisions involve new, but unsurprising information, that is at least partially consistent with the previous learned beliefs. In these circumstances, the Boltzmann network incrementally adjusts the priority of model state exemplars in accordance with preference; the traditional approach in SCNM logic. However, in the majority of situations the new information will be surprisingly inconsistent with the previous beliefs. In these circumstances, the pre-order on model states stratified by preference, will not have sufficient granularity to represent the conflicting requirements of ranking based on compositional atomic typicality. This novel experimental finding has not previously been considered in the logical conjecture on Belief Revision.

We discuss prototype formation in the Hopfield network. Typically, Hebbian learning with highly correlated states leads to degraded memory performance. We show that this type of learning can lead to prototype formation, where unlearned states emerge as representatives of large correlated subsets of states, alleviating capacity woes. This process has similarities to prototype learning in human cognition. We provide a substantial literature review of prototype learning in associative memories, covering contributions from psychology, statistical physics, and computer science. We analyze prototype formation from a theoretical perspective and derive a stability condition for these states based on the number of examples of the prototype presented for learning, the noise in those examples, and the number of nonexample states presented. The stability condition is used to construct a probability of stability for a prototype state as the factors of stability change. We also note similarities to traditional network analysis, allowing us to find a prototype capacity. We corroborate these expectations of prototype formation with experiments using a simple Hopfield network with standard Hebbian learning. We extend our experiments to a Hopfield network trained on data with multiple prototypes and find the network is capable of stabilizing multiple prototypes concurrently. We measure the basins of attraction of the multiple prototype states, finding attractor strength grows with the number of examples and the agreement of examples. We link the stability and dominance of prototype states to the energy profile of these states, particularly when comparing the profile shape to target states or other spurious states.

In learning to program and understanding how a programming language controls a computer, learners develop both insights and misconceptions whilst their mental models are gradually refined. It is important that the learner is able to distinguish the different elements and roles of a computer (compiler, interpreter, memory, etc.), which novice programmers may find difficult to comprehend. Forming accurate mental models is one of the potential sources of difficulty inextricably linked to mastering computing concepts and processes, and for learning computer programming. It is common to use some form of representation (e.g., an abstract machine or a Computational Agent (CA)) to support technical or pedagogic explanations. The Notional Machine (NM) is a pedagogical device that entails one or more computational concepts, originally described as an idealised computer operating with the constructs of a particular programming language. It can be used to support specific or general learning goals and will typically have some concrete representation that can be referred to. Computational Thinking (CT), which is defined as a way of thinking that is used for [computational] problem solving, is often presented as using a CA to carry out information processing presented by a solution. In CT, where the typical goal is to produce an algorithm or a computer program, the CA seemingly serves a purpose very similar to an NM. Although it changes through the different stages of development (of the learner and of the curriculum), the roles of CAs and NMs can be seen as versatile tools that connect a learner’s mental model with the conceptual model of a program. In this article, we look at this relationship between CAs and NMs, and indicate how they would look at different stages of learning. We traverse the range of definitions and usages of these concepts, and articulate models that clarify how these are viewed in the literature. This includes exploring the nature of machines and agents, and how historical views of these relate to modern pedagogy for computation. We argue that the CA can be seen as an abstract, simplified variant of an NM that provides a useful perspective to the learner to support them to form robust mental models of NMs more efficiently and effectively. We propose that teaching programming should make use of the idea of a CA at different stages of learning, as a link that connects a learner’s mental model to a full NM.

Ten years ago new computer science and programming achievement standards were introduced to New Zealand's national curriculum for students in their final three years at high school. However, as recent research has shown, the goal of placing these subjects on a par with subjects such as Physics and Maths has not been met. Therefore it is timely to investigate why student participation and retention for computer science and programming standards remain low. In this article, we present results from an in-depth qualitative study of students' perspectives regarding computer science and programming achievement standards. We identify eight main themes which students report as directly influencing their participation and retention in these standards. For example the placement of these standards outside the core academic areas in the curriculum, issues with the content, the assessments, and support materials as well as the guidance provided by some schools were identified as major obstacles for participation.

This paper explores a major theoretical framework from psychology, Dual Process Theory (DPT) , which has received surprisingly little attention in the computing education literature. DPT postulates the existence of two qualitatively different kinds of cognitive systems, a fast, intuitive “System 1” and a slow, reflective “System 2”. System 1 is associated with cognitive factors such as crystallized intelligence, long-term memory and associative learning; System 2 with fluid intelligence, working memory, and rule learning. This paper summarizes DPT and the way it has been expressed and explored in literatures relating to intelligence, memory, learning, attention, cognitive load, and more. It proposes a summary model, the Dual Process Cycle (DPC) . It then considers example concepts from computing education within the context of this model. Examples include programming expertise, mental models of programs, the notional machine, code reading and code writing, and the theory of Learning Edge Momentum (LEM) . In conclusion, it is argued that the DPC (and the framework of DPTs in general) provides a useful context for defining such concepts more richly and exactly, and for generating interesting questions about them.

Computer science and programming achievement standards were introduced as elective subjects in New Zealand's high schools 10 years ago, but still the uptake is very low. Fewer than 4% of students chose to take computer science and fewer than 10% took programming each year. This motivated us to explore students' reasons for not considering these subjects. Ten semi-structured interviews conducted with first year students who were enrolled in an introductory programming course resulted in eight main themes. Most important reasons for not taking the standards were a lack of guidance and the placement of those subjects outside the core academic subjects in the curriculum.

Abstract This paper looks at the representation of supra-classical, non-monotonic (SCNM) logic by an artificial neural network. It identifies the features of defeasiblity in this logic related to inference in the context of common-sense reasoning. It considers the machine characteristics that make a representation possible, with reference to previous literature. We describe a theoretical environment for investigating the representation and provide experimental evidence confirming that a Boltzmann machine is a suitable network representation. A Boltzmann machine can learn an input distribution corresponding to a preference relation and explicitly retrieve appropriate model states, constituting one-to-many mappings, entailed by the uncertain information contained in a premiss. The place of the Boltzmann machine in knowledge representation is discussed. In future papers, this neural network model of SCNM logic will serve as an experimental gateway for exploration of typicality and belief revision.

The 32 chapters of the 2019 Cambridge Handbook of Computing Education Research synthesize the existing research in computing education and propose new directions for future research. An author from each chapter will summarize their chapter with auto-advancing slides. Attendees will be introduced to the breadth of content in the new handbook and can identify chapters of interest. This fits uniquely as a special session, and will likely be informative, inspiring, and overwhelming.

This session announces the six best papers from the first five ICER conferences, as chosen by a committee of former and current ICER chairs. It explains how these papers were selected, and for each paper gives its abstract, and comments from the committee explaining why each paper is an exemplary ICER paper and contribution to the field of computing education research.

Learning the first programming language is challenging for many students. High failure rates and bimodally distributed grades lead to a pedagogical interest in supporting students in first-year programming courses (CS1). In higher education, the important role of feedback for guiding the learning process and improving the learning outcome is widely acknowledged. This article introduces contemporary models of effective feedback practice as found in the higher education literature and offers an interpretation of those in the CS1 context. One particular CS1 course and typical course components are investigated to identify likely loci for feedback interventions and to connect related computer science education literature to these forms of feedback.

For many years computing in New Zealand schools was focused on teaching students how to use computers, and there was little opportunity for students to learn about programming and computer science as formal subjects. In this article we review a series of initiatives that occurred from 2007 to 2009 that led to programming and computer science being made available formally as part of the National Certificate in Educational Achievement (NCEA), the main school-leaving assessment, in 2011. The changes were phased in from 2011 to 2013, and we review this process using the Darmstadt model, including describing the context of the school system, the socio-cultural factors in play before, during and after the changes, the nature of the new standards, the reactions and roles of the various stakeholders, and the teaching materials and methods that developed. The changes occurred very quickly, and we discuss the advantages and disadvantages of having such a rapid process. In all these changes, teachers have emerged as having a central role, as they have been key in instigating and implementing change.

In 2011 New Zealand introduced computer science into high schools after a long period when computing was mainly focussed on training students to be users. The transition was rapid, and teachers had little time to upskill to prepare for the new topics, and yet there was widespread voluntary adoption of the new standards. The role of teachers and the national teachers' organisation in making the change has been pivotal, and this paper reviews the changes from the teachers' perspective. This story is intended to inform those planning similar changes in other countries, and provide a context for the next steps in NZ. The discussion centres around a survey of 91~teachers, which reveals strong intrinsic motivation from teachers to make the changes, a mixture of prior knowledge and skills that teachers shared with each other through peer support and online communication, a low level of confidence as teachers of computer science, and a need for further professional development.

Computer science became available as a nationally assessed topic in NZ schools for the first time in 2011. We review the introduction of computer science as a formal topic, including the level of adoption, issues that have arisen in the process of introducing it, and work that has been undertaken to address those issues.

In this paper we suggest that in order to advance, the field of computer science education needs to craft its own research methods, to augment the borrowing of "traditional" methods such as semi-structured interviews and surveys from other research traditions. Two example instruments used in our recent research are discussed. We adopt the metaphor of "bricolage" to characterise not only what researchers do, but to argue that this may be a necessary step towards developing theory.

One indication of the maturation of Computer Science Education as a research-based discipline is the recent emergence of several large-scale studies spanning multiple institutions. This paper examines a "family" of these multi-institutional, multi-national studies, detailing core elements and points of difference in both study design and the organization of the research team, and highlighting the costs and benefits associated with the different approaches.

Pseudorehearsal is a mechanism proposed by Robins which alleviates catastrophic forgetting in multi-layer perceptron networks. In this paper, we extend the exploration of pseudorehearsal to a Hopfield-type net. The same general principles apply: old information can be rehearsed if it is available, and if it is not available, then generating and rehearsing approximations of old information that 'map' the behaviour of the network can also be effective at preserving the actual old information itself. The details of the pseudorehearsal mechanism, however, benefit from being adapted to the dynamics of Hopfield nets so as to exploit the extra attractors created in state space during learning. These attractors are usually described as 'spurious' or 'cross-talk', and regarded as undesirable, interfering with the retention of the trained population items. Our simulations have shown that, in another sense, such attractors can in fact be useful in preserving the learned population. In general terms, a solution to the catastrophic forgetting problem enables the on-going or sequential learning of information in artificial neural networks, and consequently also provides a framework for the modelling of lifelong learning/developmental effects in cognition.

The purpose of this paper is to review the cognitive literature regarding transfer in order to provide a context for the consideration of transfer in neural networks. We consider transfer under the three general headings of analogy, skill transfer and metaphor. The emphasis of the research in each of these areas is quite different and the literatures are largely distinct. Important common themes emerge, however, relating to the role of similarity, the importance of 'surface content' and the nature of the representations that are used. We will draw out these common themes and note ways of facilitating transfer. We also briefly note possible implications for the study of transfer in neural networks.

In this paper we explore the topic of the consolidation of information in neural network learning. One problem in particular has limited the ability of a broad range of neural networks to perform ongoing learning and consolidation. This is 'catastrophic forgetting', the tendency for new information, when it is learned, to disrupt old information. We will review and slightly extend the rehearsal and pseudorehearsal solutions to the catastrophic forgetting problem presented in Robins (1995). The main focus of this paper is to then relate these mechanisms to the consolidation processes which have been proposed in the psychological literature regarding sleep. We suggest that the catastrophic forgetting problem in artificial neural networks (ANNs) is a problem that has actually occurred in the evolution of the mammalian brain, and that the pseudorehearsal solution to the problem in ANNs is functionally equivalent to the sleep consolidation solution adopted by the brain. Finally, we review related work by McClelland et al. (1995) and propose a tentative model of learning and sleep that emphasizes consolidation mechanisms and the role of the hippocampus.

This paper reviews the problem of catastrophic forgetting (the loss or disruption of previously learned information when new information is learned) in neural networks, and explores rehearsal mechanisms (the retraining of some of the previously learned information as the new information is added) as a potential solution. We replicate some of the experiments described by Ratcliff (1990), including those relating to a simple 'recency' based rehearsal regime. We then develop further rehearsal regimes which are more effective than recency rehearsal. In particular, 'sweep rehearsal' is very successful at minimizing catastrophic forgetting. One possible limitation of rehearsal in general, however, is that previously learned information may not be available for retraining. We describe a solution to this problem, 'pseudorehearsal', a method which provides the advantages of rehearsal without actually requiring any access to the previously learned information (the original training population) itself. We then suggest an interpretation of these rehearsal mechanisms in the context of a function approximation based account of neural network learning. Both rehearsal and pseudorehearsal may have practical applications, allowing new information to be integrated into an existing network with minimum disruption of old information.

This paper proposes an extension to the basic framework of distributed representation through the learning and use of different sorts of information—“multiple representations”—in connectionist/neural network systems. In current distributed networks units are typically ascribed only one “representing” or information carrying state (activation). Similarly, connections carry a single piece of information (a weight derived from the structure of the population of patterns). In this paper we explore units and connections with multiple information carrying states. In this extended framework, multiple distributed representations can coexist with a given pattern of activation. Processing may be based on the interaction of these representations and multiple learning processes can occur simultaneously in a network. We illustrate these extensions using (in addition to patterns of activation) “centrality distribution” representations. Centrality distributions are applied to two tasks, the representation of category and type hierarchy information and the highlighting of exceptional mappings to speed up learning. We suggest that the use of multiple distributed representations in a network can increase the flexibility and power of connectionist systems while remaining within the subsymbolic paradigm. This topic is of particular relevance in the context of the recent interest in the limitations of connectionism and the interface between connectionist and symbolic methods.