Many recent advances in neural information retrieval models, which predict top-K items given a query, learn directly from a large training set of (query, item) pairs. However, they are often insufficient when there are many previously unseen (query, item) combinations, often referred to as the cold start problem. Furthermore, the search system can be biased towards items that are frequently shown to a query previously, also known as the 'rich get richer' (a.k.a. feedback loop) problem. In light of these problems, we observed that most online content platforms have both a search and a recommender system that, while having heterogeneous input spaces, can be connected through their common output item space and a shared semantic representation. In this paper, we propose a new Zero-Shot Heterogeneous Transfer Learning framework that transfers learned knowledge from the recommender system component to improve the search component of a content platform. First, it learns representations of items and their natural-language features by predicting (item, item) correlation graphs derived from the recommender system as an auxiliary task. Then, the learned representations are transferred to solve the target search retrieval task, performing query-to-item prediction without having seen any (query, item) pairs in training. We conduct online and offline experiments on one of the world's largest search and recommender systems from Google, and present the results and lessons learned. We demonstrate that the proposed approach can achieve high performance on offline search retrieval tasks, and more importantly, achieved significant improvements on relevance and user interactions over the highly-optimized production system in online experiments.

We explore content-based representation learning strategies tailored for large-scale, uncurated music collections that afford only weak supervision through unstructured natural language metadata and co-listen statistics. At the core is a hybrid training scheme that uses classification and metric learning losses to incorporate both metadata-derived text labels and aggregate co-listen supervisory signals into a single convolutional model. The resulting joint text and audio content embedding defines a similarity metric and supports prediction of semantic text labels using a vocabulary of unprecedented granularity, which we refine using a novel word-sense disambiguation procedure. As input to simple classifier architectures, our representation achieves state-of-the-art performance on two music tagging benchmarks.

Embedding based models have been the state of the art in collaborative filtering for over a decade. Traditionally, the dot product or higher order equivalents have been used to combine two or more embeddings, e.g., most notably in matrix factorization. In recent years, it was suggested to replace the dot product with a learned similarity e.g. using a multilayer perceptron (MLP). This approach is often referred to as neural collaborative filtering (NCF). In this work, we revisit the experiments of the NCF paper that popularized learned similarities using MLPs. First, we show that with a proper hyperparameter selection, a simple dot product substantially outperforms the proposed learned similarities. Second, while a MLP can in theory approximate any function, we show that it is non-trivial to learn a dot product with an MLP. Finally, we discuss practical issues that arise when applying MLP based similarities and show that MLPs are too costly to use for item recommendation in production environments while dot products allow to apply very efficient retrieval algorithms. We conclude that MLPs should be used with care as embedding combiner and that dot products might be a better default choice.

Large-scale deep learning benefits from an emerging class of AI accelerators. Some of these accelerators' designs are general enough for compute-intensive applications beyond AI and Cloud TPU is one such example. In this paper, we demonstrate a novel approach using TensorFlow on Cloud TPU to simulate the two-dimensional Ising Model. TensorFlow and Cloud TPU framework enable the simple and readable code to express the complicated distributed algorithm without compromising the performance. Our code implementation fits into a small Jupyter Notebook and fully utilizes Cloud TPU's efficient matrix operation and dedicated high speed inter-chip connection. The performance is highly competitive: it outperforms the best published benchmarks to our knowledge by 60% in single-core and 250% in multi-core with good linear scaling. When compared to Tesla V100 GPU, the single-core performance maintains a ~10% gain. We also demonstrate that using low precision arithmetic---bfloat16---does not compromise the correctness of the simulation results.

In recommender systems based on low-rank factorization of a partially observed user-item matrix, a common phenomenon that plagues many otherwise effective models is the interleaving of good and spurious recommendations in the top-K results. A single spurious recommendation can dramatically impact the perceived quality of a recommender system. Spurious recommendations do not result in serendipitous discoveries but rather cognitive dissonance. In this work, we investigate folding, a major contributing factor to spurious recommendations. Folding refers to the unintentional overlap of disparate groups of users and items in the low-rank embedding vector space, induced by improper handling of missing data. We formally define a metric that quantifies the severity of folding in a trained system, to assist in diagnosing its potential to make inappropriate recommendations. The folding metric complements existing information retrieval metrics that focus on the number of good recommendations and their ranks but ignore the impact of undesired recommendations. We motivate the folding metric definition on synthetic data and evaluate its effectiveness on both synthetic and real world datasets. In studying the relationship between the folding metric and other characteristics of recommender systems, we observe that optimizing for goodness metrics can lead to high folding and thus more spurious recommendations.

When building a recommender system, how can we ensure that all items are modeled well? Classically, recommender systems are built, optimized, and tuned to improve a global prediction objective, such as root mean squared error. However, as we demonstrate, these recommender systems often leave many items badly-modeled and thus under-served. Further, we give both empirical and theoretical evidence that no single matrix factorization, under current state-of-the-art methods, gives optimal results for each item.

The SAL cognitive architecture is a synthesis of two well-established constituents: ACT-R, a hybrid symbolic-subsymbolic cognitive architecture, and Leabra, a neural architecture. These component architectures have vastly different origins yet suggest a surprisingly convergent view of the brain, the mind and behaviour. Furthermore, both of these architectures are internally pluralistic, recognising that models at a single level of abstraction cannot capture the required richness of behaviour. In this article, we offer a brief principled defence of epistemological pluralism in cognitive science and artificial intelligence, and elaborate on the SAL architecture as an example of how pluralism can be highly effective as an approach to research in cognitive science.

As in political debate, the commentaries in this issue mostly tend towards the two ends of a similar spectrum: ‘one true path’ versus ‘anything goes’. We hope to persuade swing voters of a more mod...

This article describes the development of a real-time model-based training system that provides adaptive “over-the-shoulder” (OTS) instructions to trainees as they learn to perform an Anti-Air Warfare Coordinator (AAWC) task. The long-term goal is to develop a system that will provide real-time instructional materials based on learners’ actions, so that eventually the initial set of instructions on a task can be strengthened, complemented, or overridden at different stages of training. The training system is based on the ACT-R architecture, which serves as the theoretical background for the cognitive model that monitors the learning process of the trainee. An experiment was designed to study the impact of OTS instructions on learning. Results showed that while OTS instructions facilitated short-term learning, (a) they took time away from the processing of current information, (b) their effects tended to decay rapidly in initial stages of training, and (c) their effects on training diminished when the OTS instructions were proceduralized in later stages of training. A cognitive model that learned from both the upfront and OTS instructions was created and provided good fits to the learning and performance data collected from human participants. Our results suggest that to fully capture the symbiotic performance between humans and intelligent training systems, it is important to closely monitor the learning process of the trainee so that instructional interventions can be delivered effectively at different stages of training. We proposed that such a flexible system can be developed based on an adaptive cognitive model that provides real-time predictions on learning and performance.

The advent of second-generation intelligent computer tutors raises an important instructional design question: when should tutorial advice be presented in problem solving? This paper examines four feedback conditions in the ACT Programming Tutor. Three versions offer the student different levels of control over error feedback and correction: (a) immediate feedback and immediate error correction; (b) immediate error flagging and student control of error correction; (c) feedback on demand and student control of error correction. A fourth, No-tutor condition offers no stepby-step problem solving support. The immediate feedback group with greatest tutor control of problem solving yielded the most efficient learning. These students completed the tutor problems fastest, and the three tutor-supported groups performed equivalently on tests. Questionnaires revealed little student preference among the four conditions. These results suggest that students will need explicit guidance to benefit from learning opportunities that arise when they have greater control over tutorial assistance.

This article describes and evaluates a class of methods for performing automated analysis of eye-movement protocols. Although eye movements have become increasingly popular as a tool for investigating user behavior, they can be extremely difficult and tedious to analyze. In this article we propose an approach to automating eye-movement protocol analysis by means of tracing-relating observed eye movements to the sequential predictions of a process model. We present three tracing methods that provide fast and robust analysis and alleviate the equipment noise and individual variability prevalent in typical eye-movement protocols. We also describe three applications of the tracing methods that demonstrate how the methods facilitate the use of eye movements in the study of user behavior and the inference of user intentions.

We discuss a novel type of interface, the intelligent gaze-added interface, and describe the design and evaluation of a sample gaze-added operating-system interface. Gaze-added interfaces, like current gaze-based systems, allow users to execute commands using their eyes. However, while most gaze-based systems replace the functionality of other inputs with that of gaze, gaze-added interfaces simply add gaze functionality that the user can employ if and when desired. Intelligent gaze-added interfaces utilize a probabilistic algorithm and user model to interpret gaze focus and alleviate typical problems with eye-taking data. We extended a standard WIMP operating-system interface into a new interface, IGO, that incorporates intelligent gaze-added input. In a user study, we found that users quickly adapted to the new interface and utilized gaze effectively both alone and with other inputs.

Click-down (or pull-down) menus have long been a key component of graphical user interfaces, yet we know surprisingly little about how users actually interact with such menus. Nilsens [8] study on menu selection has led to the development of a number of models of how users perform the task [6, 21. However, the validity of these models has not been empirically assessed with respect to eye movements (though [l] presents some interesting data that bear on these models). The present study is an attempt to provide data that can help refine our understanding of how users interact with such menus.

The ACT-R system is a general system for modeling a wide range of higher level cognitive processes. Recently, it has been embellished with a theory of how its higher level processes interact with a visual interface. This includes a theory of how visual attention can move across the screen, encoding information into a form that can be processed by ACT-R. This system is applied to modeling several classic phenomena in the literature that depend on the speed and selectivity with which visual attention can move across a visual display. ACT-R is capable of interacting with the same computer screens that subjects do and, as such, is well suited to provide a model for tasks involving human-computer interaction. In this article, we discuss a demonstration of ACT-R's application to menu selection and show that the ACT-R theory makes unique predictions, without estimating any parameters, about the time to search a menu. These predictions are confirmed.

Declarative transfer from one domain to another can be observed in a systematic decrease in the time spent reading an instructional text and processing help during problem solving. Two experiments, done in the programming domain, tested the hypothesis that subjects introduced to a first programming language develop a representation of basic programming concepts that helps them integrate new declarative knowledge from a second programming language. This article shows that the effect on reading was greater for pages that were conceptually close across texts and for subjects who had fully mastered the basic concepts in the first language. A regression model of reading showed an effect on processes that are responsible for the analysis of novel words and examples, whereas general strategic reading processes remained unaffected. The increased reading speed was not accompanied by a greater understanding of the text. Effects of a common programming interface and transfer of procedural knowledge appeared to be negligible on the kind of problems considered. This study supports the distinction between procedural and declarative transfer.

This article presents a series of four experiments investigating students' debugging of LISP programs. The experiments involve a population of students who know LISP reasonably well in that their errors are best classified as slips (Brown & Van Lehn, 1980). That is, students are unlikely to repeat the same errors either within their program or across programs (Experiment 1). The students' understanding of LISP is also reflected in their debugging behavior: They can usually fix a bug once they locate it. Students' difficulties are in locating the erroneous line of code. We observe that students use a variety of bug-location strategies during debugging (Experiment 2) and that the choice of strategy differs depending on whether students are debugging their own programs or other students' programs (Experiment 3). In addition, we observe that although the different bug-location strategies affect which lines of a program are searched, once students decide on a line, their ability to judge whether or not the line is correct and their ability to correct an error are not substantially affected by the strategy used to locate the line (Experiment 4). Finally, we argue that our results have implications not only for debugging in other computer languages, but for the general processes involved in troubleshooting as well.

Two experiments studied the acquisition and transfer of text-editing skill. The first experiment, originally reported in Singley and Anderson (1985) but reanalyzed in greater detail here, found nearly total transfer between two similar line editors and partial transfer from the line editors to a screen editor. Analyses of the keystroke data revealed that the majority of the improvement during both learning and transfer was concentrated in the planning components of the skill. The second experiment found little evidence for negative transfer between a pair of screen editors designed for maximal interference using a classic interference paradigm. The few instances of negative transfer observed were better characterized as the positive transfer of nonoptimal methods rather than instances of true procedural interference. These results support an identical elements model of transfer based on a production system representation of cognitive skill. The relative magnitudes of transfer observed were consistent with detailed measures of production system overlap. In addition, localized transfer sites were hypothesized and identified through a series of microanalyses. Finally, specific transfer predictions based on the differential practice of general and specific components were tested and confirmed.

Two experiments were performed to study students' ability to write recursive and iterative programs and transfer between these two skills. Subjects wrote functions to accumulate instances into a list. Problems varied in terms of whether they were recursive or iterative, whether they operated on lists or numbers, whether they accumulated results in forward or backward manner, whether they accumulated on success or failure, and whether they simply skipped or ejected on failure to accumulate. Subjects had real difficulty only with the dimensions concerned with flow of control, namely, recursive versus iterative, and skip versus eject. We found positive transfer from writing iterative functions to writing recursive functions, but not vice versa. A subsequent protocol study revealed subjects had such a poor mental model of recursion that they developed poor learning strategies which hindered their understanding of iteration. It is argued that having an adequate model of the functionality of programming is prerequisite to learning to program, and that it is sensible pedagogical practice to base understanding of recursive flow of control on understanding iterative flow of control.

Four experiments study the errors students make using LISP functions. The first two experiments show that frequency of errors is increased by increasing the complexity of irrelevant aspects of the problem. The experiments also show that the distribution of errors is largely random and that subjects' errors seem to result from slips rather than from misconceptions. Experiment 3 shows that subjects' errors tend to involve loss of parentheses in answers when the resulting errors are well-formed LISP expressions. Experiment 4 asks subjects, who knew no LISP, to judge the reasonableness of the answers to various LISP function calls. Subjects could detect many errors on the basis of general criteria of what a reasonable answer should look like. On the basis of these four experiments, we conclude that errors occur when there is a loss of information in the working memory representation of the problem and when the resulting answer still looks reasonable.

Ferroelectric shift registers having completely independent parallel or serial inputs and outputs have been designed and constructed. The principal components of these shift registers are single crystals of barium titanate and silicon junction diodes. Two ferroelectric units and two to three silicon junction diodes are required for each stage of the shift register. Practical operating speeds for 10-stage shift registers with transistor drives are at present from 0 to 5 kc. The small size of the ferroelectric units and the low power consumption in this speed range make the ferroelectric shift register attractive for many digital circuit applications.