Language models that are trained on the nextword prediction task have been shown to accurately model human behavior in word prediction and reading speed.In contrast with these findings, we present a scenario in which the performance of humans and LMs diverges.We collected a dataset of human next-word predictions for five stimuli that are formed by repeating spans of text.Human and GPT-2 LM predictions are strongly aligned in the first presentation of a text span, but their performance quickly diverges when memory (or in-context learning) begins to play a role.We traced the cause of this divergence to specific attention heads in a middle layer.Adding a power-law recency bias to these attention heads yielded a model that performs much more similarly to humans.We hope that this scenario will spur future work in bringing LMs closer to human behavior.1

Richard Antonello, Nicole Beckage, Javier Turek, Alexander Huth. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

How related are the representations learned by neural language models, translation models, and language tagging tasks? We answer this question by adapting an encoder-decoder transfer learning method from computer vision to investigate the structure among 100 different feature spaces extracted from hidden representations of various networks trained on language tasks.This method reveals a low-dimensional structure where language models and translation models smoothly interpolate between word embeddings, syntactic and semantic tasks, and future word embeddings. We call this low-dimensional structure a language representation embedding because it encodes the relationships between representations needed to process language for a variety of NLP tasks. We find that this representation embedding can predict how well each individual feature space maps to human brain responses to natural language stimuli recorded using fMRI. Additionally, we find that the principal dimension of this structure can be used to create a metric which highlights the brain's natural language processing hierarchy. This suggests that the embedding captures some part of the brain's natural language representation structure.

Recent work has shown that topological enhancements to recurrent neural networks (RNNs) can increase their expressiveness and representational capacity. Two popular enhancements are stacked RNNs, which increases the capacity for learning non-linear functions, and bidirectional processing, which exploits acausal information in a sequence. In this work, we explore the delayed-RNN, which is a single-layer RNN that has a delay between the input and output. We prove that a weight-constrained version of the delayed-RNN is equivalent to a stacked-RNN. We also show that the delay gives rise to partial acausality, much like bidirectional networks. Synthetic experiments confirm that the delayed-RNN can mimic bidirectional networks, solving some acausal tasks similarly, and outperforming them in others. Moreover, we show similar performance to bidirectional networks in a real-world natural language processing task. These results suggest that delayed-RNNs can approximate topologies including stacked RNNs, bidirectional RNNs, and stacked bidirectional RNNs – but with equivalent or faster runtimes for the delayed-RNNs.

Natural language contains information at multiple timescales. To understand how the human brain represents this information, one approach is to build encoding models that predict fMRI responses to natural language using representations extracted from neural network language models (LMs). However, these LM-derived representations do not explicitly separate information at different timescales, making it difficult to interpret the encoding models. In this work we construct interpretable multi-timescale representations by forcing individual units in an LSTM LM to integrate information over specific temporal scales. This allows us to explicitly and directly map the timescale of information encoded by each individual fMRI voxel. Further, the standard fMRI encoding procedure does not account for varying temporal properties in the encoding features. We modify the procedure so that it can capture both short- and long-timescale information. This approach outperforms other encoding models, particularly for voxels that represent long-timescale information. It also provides a finer-grained map of timescale information in the human language pathway. This serves as a framework for future work investigating temporal hierarchies across artificial and biological language systems.

The field of machine programming (MP), the automation of the development of software, is making notable research advances. This is, in part, due to the emergence of a wide range of novel techniques in machine learning. In this paper, we apply MP to the automation of software performance regression testing. A performance regression is a software performance degradation caused by a code change. We present AutoPerf – a novel approach to automate regression testing that utilizes three core techniques: (i) zero-positive learning, (ii) autoencoders, and (iii) hardware telemetry. We demonstrate AutoPerf’s generality and efficacy against 3 types of performance regressions across 10 real performance bugs in 7 benchmark and open-source programs. On average, AutoPerf exhibits 4% profiling overhead and accurately diagnoses more performance bugs than prior state-of-the-art approaches. Thus far, AutoPerf has produced no false negatives.

Geodesic distance matrices can reveal shape properties that are largely invariant to non-rigid deformations, and thus are often used to analyze and represent 3-D shapes. However, these matrices grow quadratically with the number of points. Thus for large point sets it is common to use a low-rank approximation to the distance matrix, which fits in memory and can be efficiently analyzed using methods such as multidimensional scaling (MDS). In this paper we present a novel sparse method for efficiently representing geodesic distance matrices using biharmonic interpolation. This method exploits knowledge of the data manifold to learn a sparse interpolation operator that approximates distances using a subset of points. We show that our method is 2x faster and uses 20x less memory than current leading methods for solving MDS on large point sets, with similar quality. This enables analyses of large point sets that were previously infeasible.

Cognitive neuroscience seeks to explain the organization of the brain, but typically focuses on aspects that are shared across people rather than those that vary across individuals. Here, we present a new method for analyzing brain imaging data that captures both shared and individual components of brain activity. Inspired by the shared response model (SRM) and the robust principal components analysis, the robust shared response model (RSRM) aligns functional topographies across humans while preserving a component of sparse, individual activity. Experimental results on adult data showed that RSRM performs as well as or better than SRM, while at the same time capturing reliable markers of individual variability. In a test case of participants with extreme variability, we found that RSRM was able to improve the accuracy more than 60% over SRM for the coding of infant fMRI data.

Practical limitations on the duration of individual fMRI scans have led neuroscientist to consider the aggregation of data from multiple subjects. Differences in anatomical structures and functional topographies of brains require aligning data across subjects. Existing functional alignment methods serve as a preprocessing step that allows subsequent statistical methods to learn from the aggregated multi-subject data. Despite their success, current alignment methods do not leverage the labeled data used in the subsequent methods. In this work we propose a semi-supervised scheme that simultaneously learns the alignment and performs the analysis. We derive a specific instance of the scheme using the Shared Response Model for alignment and Multinomial Logistic Regression for classification. In our experiments this method improves the average classification accuracy from 65.5% to 68.5%, and from 5.3% to 6.1% over the independently-trained methods. Furthermore, our method achieves similar prediction with almost half the samples used for alignment.

The scale of functional magnetic resonance image data is rapidly increasing as large multi-subject datasets are becoming widely available and high-resolution scanners are adopted. The inherent low-dimensionality of the information in this data has led neuroscientists to consider factor analysis methods to extract and analyze the underlying brain activity. In this work, we consider two recent multi-subject factor analysis methods: the Shared Response Model and the Hierarchical Topographic Factor Analysis. We perform analytical, algorithmic, and code optimization to enable multi-node parallel implementations to scale. Single-node improvements result in 99χ and 2062x speedups on the two methods, and enables the processing of larger datasets. Our distributed implementations show strong scaling of 3.3x and 5.5χ respectively with 20 nodes on real datasets. We demonstrate weak scaling on a synthetic dataset with 1024 subjects, equivalent in size to the biggest fMRI dataset collected until now, on up to 1024 nodes and 32,768 cores.

In ultrasound, second harmonic imaging is usually preferred due to the higher clutter artifacts and speckle noise common in the first harmonic image. Typical ultrasound use either one or the other image, applying corresponding filters for each case. In this work we propose a method based on a joint sparsity model that fuses the first and second harmonic images while performing clutter mitigation and noise reduction. Our approach, Fused Morphological Component Analysis (FMCA), uses two adaptive dictionaries for characterizing the clutter components in each image, and a common dictionary for the tissue representation. Our results indicate that the obtained images contain less clutter artifacts, less speckle noise and as such enjoy of the benefits of both harmonic input images.

The transition to multicore architectures creates significant challenges for programming systems. Taking advantage of specialized processing cores such as those in the Cell BE processor and managing all the required data movement inside the processor cannot be done efficiently without help from the software infrastructure. Alongside new programming models and compiler support for multicores, programmers need performance evaluation and analysis tools. In this paper, we present tools that help analyze the performance of applications executing on the Cell platform. The performance debugging tool (PDT) provides a means for recording significant events during program execution, maintaining the sequential order of events, and preserving important runtime information such as core assignment and relative timing of events. The trace analyzer (TA) reads and visualizes the PDT traces. We describe the architecture of the PDT and present several important use cases demonstrating the usage of PDT and TA to understand the performance of several workloads. We also discuss the overhead of tracing and its impact on the benchmark execution and performance analysis.