Undergraduate research experiences are often limited to small-scale apprenticeship models, leaving many students without accessible entry points into research practice. This paper presents the design and evaluation of a semester-long course for undergraduates to gain research experience in Machine Learning. The course, led by one faculty instructor, enables nearly a hundred students to engage in structured research through a scaffolded replication-and-extension project, where students first replicate a published research project and then implement novel additions. The course integrates instructional modules (e.g., guided paper reading, proposal writing, public presentation) with project milestones (e.g., replication, extension, poster) to support research learning for students with diverse backgrounds. Every component of research is visited several times, with each iteration having progressively increased autonomy coupled with simultaneously decreased scaffolding. We find that the scaffolding modules help students develop foundational conceptual and procedural understanding of doing research, and the project milestones on replication and extension help them gain execution skills gradually. Students also report developing a researcher mindset and feeling like they understand the research process better. We discuss the principles used to design a scalable research-based course: balancing scaffolding to provide foundational understanding with autonomy for students to “feel like real researchers”.

When crowdsourcing the creation of machine learning datasets, statistical distributions that capture diverse answers can represent ambiguous data better than a single best answer. Unfortunately, collecting distributions is expensive because a large number of responses need to be collected to form a stable distribution. Despite this, the efficient collection of answer distributions-that is, ways to use less human effort to collect estimates of the eventual distribution that would be formed by a large group of responses-is an under-studied topic. In this paper, we demonstrate that this type of estimation is possible and characterize different elicitation approaches to guide the development of future systems. We investigate eight elicitation approaches along two dimensions: annotation granularity and estimation perspective. Annotation granularity is varied by annotating i) a single "best" label, ii) all relevant labels, iii) a ranking of all relevant labels, or iv) real-valued weights for all relevant labels. Estimation perspective is varied by prompting workers to either respond with their own answer or an estimate of the answer(s) that they expect other workers would provide. Our study collected ordinal annotations on the emotional valence of facial images from 1,960 crowd workers and found that, surprisingly, the most fine-grained elicitation methods were not the most accurate, despite workers spending more time to provide answers. Instead, the most efficient approach was to ask workers to choose all relevant classes that others would have selected. This resulted in a 21.4% reduction in the human time required to reach the same performance as the baseline (i.e., selecting a single answer with their own perspective). By analyzing cases in which finer-grained annotations degraded performance, we contribute to a better understanding of the trade-offs between answer elicitation approaches. Our work makes it more tractable to use answer distributions in large-scale tasks such as ML training, and aims to spark future work on techniques that can efficiently estimate answer distributions.

We consider the design of prediction market mechanisms known as automated market makers. We show that we can design these mechanisms via the mold of exponential family distributions, a popular and well-studied probability distribution template used in statistics. We give a full development of this relationship and explore a range of benefits. We draw connections between the information aggregation of market prices and the belief aggregation of learning agents that rely on exponential family distributions. We develop a natural analysis of the market behavior as well as the price equilibrium under the assumption that the traders exhibit risk aversion according to exponential utility. We also consider similar aspects under alternative models, such as budget-constrained traders.