Paper prototyping presents a low-entry barrier method to engaging youth in interaction design. Purely paper-based designs leave a large gap between ideation and implementation. Paper Playground is a prototyping tool that connects physical and virtual papers with JavaScript programs, enabling the creation of multimodal prototypes in both face-to-face and virtual settings. Paper Playground is being designed and developed through iterative co-design activities including youth and adults. Here we present findings from remote co-design sessions with youth, investigating what affordances the participants requested from a multimodal prototyping tool. We reflect on the co-designers desires and remarks on paper use for interactive project design, remote collaborative use, and extensibility for physical computing.

Sensory extensions enhance our awareness by transforming variations in stimuli normally undetectable by human senses into perceivable outputs. Similarly, interactive simulations for learning promote an understanding of abstract phenomena. Combining sensory extension devices with interactive simulations gives users the novel opportunity to connect their sensory experiences in the physical world to computer-simulated concepts. We explore this opportunity by designing a suite of wearable sensory extension devices that interface with a uniquely inclusive PhET Simulation, Ratio and Proportion. In this simulation, two hands can be moved on-screen to various values, representing different mathematical ratios. Users explore changing hand heights to find and maintain ratios through visual and auditory feedback. Our sensory extension devices translate force, distance, sound frequency, and magnetic field strength to quantitative values in order to control individual hands in the computer simulation. This paper describes the design of the devices and our analysis of feedback from 23 high-school aged youth who used our designs to interact with the Ratio and Proportion simulation.

Design thinking is an approach to educational curriculum that builds empathy, encourages ideation, and fosters active problem solving through hands-on design projects. Embedding participatory “co-design” into design thinking curriculum offers students agency in finding solutions to real-world design challenges, which may support personal empowerment. An opportunity to explore this prospect arose in the design of sounds for an accessible interactive science-education simulation in the PhET Project. Over the course of three weeks, PhET researchers engaged blind and visually-impaired high-school students in a design thinking curriculum that included the co-design of sounds and auditory interactions for the Balloons and Static Electricity (BASE) sim. By the end of the curriculum, students had iterated through all aspects of design thinking and performed a quantitative evaluation of multiple sound prototypes. Furthermore, the group’s mean self-efficacy rating had increased. We reflect on our curriculum and the choices we made that helped enable the students to become authentic partners in sound design.

Auditory description display is verbalized text typically used to describe live, recorded, or graphical displays to support access for people who are blind or visually impaired. Significant prior research has resulted in guidelines for auditory description for non-interactive or minimally interactive contexts. A lack of auditory description for complex interactive environments remains a tremendous barrier to access for people with visual impairments. In this work, we present a systematic design framework for designing auditory description within complex interactive environments. We illustrate how modular descriptions aligned with this framework can result in an interactive storytelling experience constructed through user interactions. This framework has been used in a set of published and widely used interactive science simulations, and in its generalized form could be applied to a variety of contexts.

Science simulations are widely used in classrooms to support inquiry-based learning of complex science concepts. These tools typically rely on interactive visual displays to convey relationships. Auditory displays, including verbal description and sonification (non-speech audio), combined with alternative input capabilities, may provide an enhanced experience for learners, particularly learners with visual impairment. We completed semi-structured interviews and usability testing with eight adult learners with visual impairment for two audio-enhanced simulations. We analyzed trends and edge cases in participants' interaction patterns, interpretations, and preferences. Findings include common interaction patterns across simulation use, increased efficiency with second use, and the complementary role that description and sonification play in supporting learning opportunities. We discuss how these control and display layers work to encourage exploration and engagement with science simulations. We conclude with general and specific design takeaways to support the implementation of auditory displays for accessible simulations.

Teaching students with learning disabilities about design thinking can prepare them to be active co-designers of learning tools and resources that will ultimately benefit them and their peers. In this paper, we outline an introductory design thinking activity conducted with students with learning disabilities and share two specific and contrasting student interactions that occurred during the activity. The two interactions highlight how being able to engage in open, respectful, and constructive idea sharing can lead to a more sophisticated and evolved design prototype. Student collaboration observed also provides insight into improved ways to scaffold learners in introductions to design thinking. We share lessons learned and ideas for how to modify this activity to better support a positive introduction to design thinking experience.

We present a multimodal science simulation, including visual and auditory (descriptions, sound effects, and sonifications) display. The design of each modality is described, as well as evaluation with learners with and without visual impairments. We conclude with challenges and opportunities at the intersection of multiple modalities.

Interactive simulations are increasingly important in science education, yet most are inaccessible to blind learners. We demonstrate an accessible version of a simulation, Balloons and Static Electricity, that illustrates responses to key challenges in providing screen reader support: the need to describe unpredictable sequences of events, the manipulation of objects that act as both controls and displays, and the management of descriptions of changes in the state of the simulation as well as of the state of the interactive object, itself. Meeting these challenges requires extending current practices for verbal description of visual interactive content.

Interactive simulations are becoming increasingly important in education, but these tools are not accessible to many learners. Here, we share some goals to guide the development of accessible simulations, and summarize early findings in development efforts. We also highlight challenges and opportunities where the unique expertise of the ASSETS community is needed. We hope to interest readers in participating in the growing community designing, developing, and researching interactive simulations for learning.