The recent proliferation of research into transformer based natural language processing has led to a number of studies which attempt to detect the presence of human-like cognitive behavior in the models. We contend that, as is true of human psychology, the investigation of cognitive behavior in language models must be conducted in an appropriate population of an appropriate size for the results to be meaningful. We leverage work in uncertainty estimation in a novel approach to efficiently construct experimental populations. The resultant tool, PopulationLM, has been made open source. We provide theoretical grounding in the uncertainty estimation literature and motivation from current cognitive work regarding language models. We discuss the methodological lessons from other scientific communities and attempt to demonstrate their application to two artificial population studies. Through population based experimentation we find that language models exhibit behavior consistent with typicality effects among categories highly represented in training. However, we find that language models don't tend to exhibit structural priming effects. Generally, our results show that single models tend to over estimate the presence of cognitive behaviors in neural models.

Currently there are very few AI tools available to conference reviewers to support the academic process. In this paper we present a naive, first attempt at an automated conference reviewer called pReview. pReview provides a paper summary, contribution summary, writing quality analysis, and set of potential related works as well as an intellectual space visualization to aid the busy academics who volunteer their time as conference paper reviewers. To limit the plethora of ethical issues emerging when a computer is left to judge a human creation, pReview and future systems of this variety should be judiciously viewed as review support tools and never as a replacement for the traditional review process.

Online learners sometimes prefer to download course content rather than view it on a course website. These students often miss out on interactive content. Knowing who downloads course materials, and why, can help course creators design courses that fit the needs of their students. In this paper we explore downloading behavior by looking at lecture videos in three online classes. We found that the number of days since a video was posted had the strongest relationship with downloading, and non-technical considerations, such as typical classroom size in a student's home country, matter more than technical issues, such as internet speed. Our findings suggest that more materials will be downloaded when a course will be available for limited time, students are less familiar with the language of instruction, students are used to classrooms with a high student-teacher ratio, or a student's internet speed is slow. Possible reasons for these relationships are discussed.

Computing for social good has become a common topic in computing circles, with professional organizations and conferences sponsoring discussions on the relevance of "social good" material for computer science research [e.g., 1] and for education [e.g., 12,13,18].

This paper and summary talk broadly survey computational sustainability research. Rather than a detailed treatment of the research projects in the area, which is beyond the scope of the paper and talk, the paper includes a meta-survey, pointing to edited collections and overviews in the literature for the interested reader. Computational sustainability research has been broadly characterized by AI methods employed, sustainability areas addressed, and contributions made to (typically, human) decision-making. The paper addresses these characterizations as well, which will facilitate a deeper synthesis later, to include the potential for developing sophisticated and holistic AI decision-making and advisory agents.

Extant research suggests that learner engagement in discussion forums is positively correlated with learner performance. In this paper we investigate which types of forum engagement are most strongly associated with final performance in MOOC courses. In particular, we compare the correlation between course final grade and two types of learner engagement: direct measures, which count the number of interactions, and indirect measures, which capture learners position in a social network. We found that direct measures have stronger correlations with final grade. However, in preliminary analyses, we also found that course instructors score higher than learners on some indirect measures. We discuss the implications of these findings and our plans for developing the work further in the future.

Course syllabi, exercises, assignments, and project specifications are creative acts of design. But there are no conventions for citing such works, as one would cite a research paper or patent. There are several potentially important reasons for establishing conventions for citing educational content, and thus tracking their diffusion and influence. (1) Design of courseware is at least as important to teaching as is delivering lectures and grading assessments. If citation conventions were established, then the degree of adoption by others could be part of a teaching-faculty member's professional evaluation. (2) Research grants often include education and outreach plans, and being able to track the influence of materials under such plans would be valuable for funding agencies and principal investigators. (3) Establishing citation conventions may raise consciousness about citing any and all creative works that an instructor uses, particularly when we want our students to embrace citation best practices. After all, what most undergraduates will see of the scholarship modeled by faculty will be through teaching. (4) Conferences like SIGCSE are increasingly competitive, and publishing a paper that reports on educational material, thereby enabling "conventional citation" may be difficult. But we still want to cite and track uptake of the educational material. This lightning talk argues that citation conventions be adapted for educational materials, and describes a sample convention. The goal of the lightening talk is to gauge interest and solicit collaborators.

We are exploring the effects of social incentives and motivation on learner performance in a massive open online course. In the preliminary study that we report here, we asked learners if they wanted to be considered for a community TAship in a subsequent offering of the course, if they finished in the top 20% of those who completed the current course instance. We prompted students near the beginning of the course and in the middle of the course. This prompt appears to have had a significant, albeit small effect on learner completion when given early in the course. The prompt had no significant effect when given later in the course. We also discuss our plans to follow-up this study.

In this paper, we present early research evaluating the predictive power of a variety of temporal features across student subpopulations with distinctive behaviors at the beginning of the course. Initial results illustrate that these features predict important differences across the subpopulations and over time in the courses. Ultimately, these results have implications for effectively targeting adaptive scaffolding tailored to the particular intentions and goals of subpopulations in MOOCs.

Computer science instructors are incorporating Massive Open Online Courses (MOOCs), and MOOC material, into their campus courses. The closed instance is the dominant model being followed, in which MOOC material (e.g., in AI, in Programming Languages, in Software Engineering) is ported to a protected LMS so that campus students can use the MOOC video and other material (e.g., assessments) and infrastructure (e.g., autograders) in a "private" manner. In an embedded instance, campus students take a live MOOC as one component of their on-campus course. Another model, albeit nascent, is to exploit self-paced open online courses so that campus cohorts can move through a course when a campus instructor wants, and at a pace the instructor wants, but still availing campus students to the advantages of interacting with global students (and vice versa). The BoF session will allow participants to talk about MOOC-based blended models, form a group interested to archiving experience, synthesizing best practices, creating shared learning-object repositories (e.g., of active learning, in-class activities), and to discuss issues of policy (e.g., FERPA).

The Markov blanket of a target variable is the minimum conditioning set of variables that makes the target independent of all other variables. Markov blankets inform feature selection, aid in causal discovery and serve as a basis for scalable methods of constructing Bayesian networks. We apply decision tree induction to the task of Markov blanket identification. Notably, we compare (a) C5.0, a widely used algorithm for decision rule induction, (b) C5C, which post-processes C5.0 's rule set to retain the most frequently referenced variables and (c) PC, a standard method for Bayesian network induction. C5C performs as well as or better than C5.0 and PC across a number of data sets. Our modest variation of an inexpensive, accurate, off-the-shelf induction engine mitigates the need for specialized procedures, and establishes baseline performance against which specialized algorithms can be compared.

Most rule learning systems posit hard decision boundaries for continuous attributes and point estimates of rule accuracy, with no measures of variance, which may seem arbitrary to a domain expert. These hard boundaries/points change with small perturbations to the training data. Moreover, rule induction typically produces a large number of rules that must be filtered and interpreted by an analyst. We describe a method of combining rules over multiple bootstrap replications of rule induction so as to reduce the total number of rules presented to an analyst and to provide measures of variance to continuous attribute decision boundaries and accuracy-point estimates. The method is illustrated with perioperative data.

Affordable, fast computers with large memories have lessened the demand for program efficiency, but applications such as browsing and searching very large databases often have ratelimiting constraints and therefore benefit greatly from improvements in efficiency.This paper empirically evaluates several variants of a common k-dimensional tree technique to demonstrate how different algorithm options influence search cost for nearest neighbors.

We briefly review each paper of the Fourteenth International Conference on Machine Learning, along with some general observations on the conference as a whole. The major topics of papers include data reduction, feature selection, ensembles of classifiers, natural language learning, text categorization, inductive logic programming, stochastic models, and reinforcement learning.

The data exploration task can be divided into three interrelated subtasks: 1) feature selection, 2) discovery, and 3) interpretation. This paper describes an unsupervised discovery method with biases geared toward partitioning objects into clusters that improve interpretability. The algorithm ITERATE employs: 1) a data ordering scheme and 2) an iterative redistribution operator to produce maximally cohesive and distinct clusters. Cohesion or intraclass similarity is measured in terms of the match between individual objects and their assigned cluster prototype. Distinctness or interclass dissimilarity is measured by an average of the variance of the distribution match between clusters. The authors demonstrate that interpretability, from a problem-solving viewpoint, is addressed by the intraclass and interclass measures. Empirical results demonstrate the properties of the discovery algorithm and its applications to problem solving.

Clustering is often used for discovering structure in data. Clustering systems differ in the objective function used to evaluate clustering quality and the control strategy used to search the space of clusterings. Ideally, the search strategy should consistently construct clusterings of high quality, but be computationally inexpensive as well. In general, we cannot have it both ways, but we can partition the search so that a system inexpensively constructs a `tentative' clustering for initial examination, followed by iterative optimization, which continues to search in background for improved clusterings. Given this motivation, we evaluate an inexpensive strategy for creating initial clusterings, coupled with several control strategies for iterative optimization, each of which repeatedly modifies an initial clustering in search of a better one. One of these methods appears novel as an iterative optimization strategy in clustering contexts. Once a clustering has been constructed it is judged by analysts -- often according to task-specific criteria. Several authors have abstracted these criteria and posited a generic performance task akin to pattern completion, where the error rate over completed patterns is used to `externally' judge clustering utility. Given this performance task, we adapt resampling-based pruning strategies used by supervised learning systems to the task of simplifying hierarchical clusterings, thus promising to ease post-clustering analysis. Finally, we propose a number of objective functions, based on attribute-selection measures for decision-tree induction, that might perform well on the error rate and simplicity dimensions.