The ability to reproduce results is important for future experimenters building on existing work. We argue that the ability to do so in a cost-effective manner – what we call practical reproducibility – should be a mainstream method of scientific exploration and as a community we should invest in building an ecosystem of tools that would support it. Since computational research artifacts require some form of computing to interpret, we argue that such ecosystem is likely to be tied to shared experimental infrastructure where unique resources can be found and effectively shared. We propose three interrelated services, solving the problems of packaging for reuse, findability, and accessibility, respectively. We describe how we developed these services in Chameleon, an NSF-funded testbed for computer science research which has supported 8,000+ users, and discuss their strengths and limitations.

Investigating phenomena that require continuous collection of data from a large and widely distributed array of hard-to-reach sources – such as understanding the performance of end-user broadband – has traditionally been hard. The opportunities in this sphere are now changing with easy availability of single board computers (SBCs), that are reliable and cheap, and thus can in principle be deployed in places of interest at large scales to gain coverage yielding statistically significant results. The challenge that this idea raises is how to create, deploy, an operate this type of infrastructure, given that its makup and deployment properties are very different from hardware deployed in datacenters. This paper presents the design of FLOTO, an observational instrument that supports the deployment and operation of mainstream SBCs to collect data through large-scale deployments in the field. FLOTO allows users to deploy devices to collect data of interest; operate those devices securely in remote locations without physical access to device; supports multi-tenant sharing of devices between different data collecting applications and user groups, that makes it possible to adapt or re-purpose the observational function of the instrument; and provides data collection and sharing functionality allowing user communities to benefit from the collected data. We describe the design of FLOTO and present a case study of its deployment as an observational instrument to collect broadband data. We conclude by discussing the possible adaptations of this instrument to study different scientific questions.