With artificial intelligence (AI) becoming more present in ed- ucation globally, it is essential to consider how cultural con- texts shape teachers’ perspectives, an understanding that sup- ports more inclusive and sustainable learning systems. This study draws on the African philosophy of Ubuntu to frame our cross-cultural investigation of how children conceptu- alize AI through the lens of their teachers. We conducted semi-structured interviews with twelve middle school teach- ers in Nigeria and the United States, asking them to interpret AI-themed essays written by students. These teacher reflec- tions revealed differing educational priorities, cultural val- ues, and infrastructural realities: U.S. educators’ interpreta- tions centered on personal development and future careers, while Nigerian teachers highlighted students’ focus on fam- ily, community well-being, and practical societal challenges. Nigerian participants also pointed to the need for improved infrastructure (e.g., electricity, internet), broader AI literacy, and education policies that reflect local needs. Our findings il- lustrate how culturally grounded worldviews, such as Ubuntu, shape interpretations of AI and its role in society, and sug- gest that AI education is never culturally neutral. We argue that AI literacy initiatives must be designed not only to teach technical skills but also to support educational sustainability, defined here as inclusive, resilient, and culturally responsive learning systems capable of evolving within diverse contexts. We offer actionable recommendations for the HCI commu- nity to co-design AI education tools that foreground collec- tive well-being, foster global digital citizenship, and reduce epistemic exclusion in the development of future technolo- gies.

As generative AI (genAI) rapidly enters classrooms, accompanied by district-level policy rollouts and industry-led teacher trainings, it is important to rethink the canonical “adopt and train” playbook. Decades of educational technology research show that tools promising personalization and access often deepen inequities due to uneven resources, training, and institutional support. Against this backdrop, we conducted semi-structured interviews with 22 teachers from a large U.S. school district that was an early adopter of genAI. Our findings reveal the motivations driving adoption, the factors underlying resistance, and the boundaries teachers negotiate to align genAI use with their values. We further contribute by unpacking the sociotechnical dynamics—including district policies, professional norms, and relational commitments—that shape how teachers navigate the promises and risks of these tools.

This article examines the ways secondary computer science and English Language Arts teachers in urban, suburban, and semi-rural schools adapted a project-based AI ethics curriculum to make it better fit their local contexts. AI ethics is an urgent topic with tangible consequences for youths’ current and future lives, but one that is rarely taught in schools. Few teachers have formal training in this area as it is an emerging field even at the university level. Exploring AI ethics involves examining biases related to race, gender, and social class, a challenging task for all teachers, and an unfamiliar one for most computer science teachers. It also requires teaching technical content which falls outside the comfort zone of most humanities teachers. Although none of our partner teachers had previously taught an AI ethics project, this study demonstrates that their expertise and experience in other domains played an essential role in providing high quality instruction. Teachers designed and redesigned tasks and incorporated texts and apps to ensure the AI ethics project would adhere to district and department level requirements; they led equity-focused inquiry in a way that both protected vulnerable students and accounted for local cultures and politics; and they adjusted technical content and developed hands-on computer science experiences to better challenge and engage their students. We use Mishra and Kohler’s TPACK framework to highlight the ways teachers leveraged their own expertise in some areas, while relying on materials and support from our research team in others, to create stronger learning experiences.

This paper presents a new approach to engaging children in Nigeria to share their views of AI. This approach is centered on an inclusive writing contest for children in a secondary school in Abuja to write about AI to compete for prizes and share their writings with others. A preliminary analysis of the first 11 articles we received exhibits diverse gender and ethnic representation that conveys cultural values and perspectives distinct from those of the children in Western countries. This finding suggests future work to conduct an in-depth cross-cultural analysis of the articles and to replicate similar writing contests to engage children in other underrepresented countries.

Empathy for children is critical for designing AI technologies that may affect children. This paper presents the work in progress of a study on the feasibility of a new method to provide objective understanding of people’s empathy for children based on functional near infrared spectroscopy (fNIRS). Adult participants (n=13) were presented with benign or concerning scenarios involving children interacting with AI technologies. Their brain activation patterns were recorded and analyzed. Preliminary data analysis revealed distinctive patterns in the mPFC region, which justifies future work to fully realize the potential of this method.

Reflection is a metacognitive skill that’s essential to creative discovery. As we design interactive technologies for reflection, how might we measure the impact of our designs? In this paper, we develop a coding scheme to explore reflective moments in the speech and language of young children during child-computer interaction. Using cross-disciplinary theories — from the learning sciences to cognitive neuroscience — we define and describe 13 reflective processes occurring within Baumer’s 3 conceptual dimensions of reflection. We then use this framework to measure the impact of a child-robot storytelling interaction with twelve children ages 4–5, and offer developmentally-appropriate transcript examples for each of the 13 reflective processes. This coding scheme provides a practical tool for exploring the impact of our designs on reflection, and can be used to guide design iteration.

Today’s cybersecurity and AI technologies are often fraught with ethical challenges. One promising direction is to teach cybersecurity and AI ethics to today’s youth. However, we know little about how these subjects are taught before college. Drawing from interviews of US high school teachers (n=16) and students (n=11), we find that cybersecurity and AI ethics are often taught in non-technical classes such as social studies and language arts. We also identify relevant topics, of which epistemic norms, privacy, and digital citizenship appeared most often. While teachers leverage traditional and novel teaching strategies including discussions (treating current events as case studies), gamified activities, and content creation, many challenges remain. For example, teachers hesitate to discuss current events out of concern for appearing partisan and angering parents; cyber hygiene instruction appears very ineffective at educating youth and promoting safer online behavior; and generational differences make it difficult for teachers to connect with students. Based on the study results, we offer practical suggestions for educators, school administrators, and cybersecurity practitioners to improve youth education on cybersecurity and AI ethics.

Tactile images are an important mode of information for the blind and low vision community but many of the common methods have high acquisition and material costs. This design inquiry looks at the use of a Cricut cutting machine to layer cardstock for creating tangible pictures and coloring pages. This report discusses the development of the technique followed by the corresponding production considerations, design iterations, pilot feedback, and limitations. We believe that in certain contexts this technique could be a viable method for conveying tactile information, especially in the context of early learning activities where practice is required for gaining life-long tactile graphicacy skills that can translate to reading maps and information graphics later in life. Additionally, the low acquisition cost of this method has value to parents and educators of blind and low vision children as a cheap DIY way to make tactile content.

When young children create, they are exploring their emerging skills. And when young children reflect, they are transforming their learning experiences. Yet early childhood play environments often lack toys and tools to scaffold reflection. In this work, we design a stuffed animal robot to converse with young children and prompt creative reflection through open-ended storytelling. We also contribute six design goals for child-robot interaction design. In a hybrid Wizard of Oz study, 33 children ages 4-5 years old across 10 U.S. states engaged in creative play then conversed with a stuffed animal robot to tell a story about their creation. By analyzing children’s story transcripts, we discover four approaches that young children use when responding to the robot’s reflective prompting: Imaginative, Narrative Recall, Process-Oriented, and Descriptive Labeling. Across these approaches, we find that open-ended child-robot interaction can integrate personally meaningful reflective storytelling into diverse creative play practices.

Recent studies show the effectiveness of interview chatbots for information elicitation. However, designing an effective interview chatbot is non-trivial. Few tools exist to help designers design, evaluate, and improve an interview chatbot iteratively. Based on a formative study and literature reviews, we propose a computational framework for quantifying the performance of interview chatbots. Incorporating the framework, we have developed iChatProfile, an assistive chatbot design tool that can automatically generate a profile of an interview chatbot with quantified performance metrics and offer design suggestions for improving the chatbot based on such metrics. To validate the effectiveness of iChatProfile, we designed and conducted a between-subject study that compared the performance of 10 interview chatbots designed with or without using iChatProfile. Based on the live chats between the 10 chatbots and 1349 users, our results show that iChatProfile helped the designers build significantly more effective interview chatbots, improving both interview quality and user experience.

Design tools and research regarding laser-cut architectures have been widely explored in the past decade. However, such discussion has mostly revolved around technical and structural design questions instead of another essential element of laser-cut models — assembly — a process that relies heavily on components’ visual affordance, therefore less accessible to blind or low vision (BLV) people. To narrow the gap in this area, we co-designed with 7 BLV people to examine their assembly experience with different laser-cut architectures. From their feedback, we proposed several design heuristics and guidelines for Daedalus, a generative design tool that can produce tactile aids for laser-cut assembly given a few high-level manual inputs. We validate the proposed aids in a user study with 8 new BLV participants. Our results revealed that BLV users can manage laser-cut assembly more efficiently with Daedalus. Going forth from this design iteration, we discuss implications for future research on accessible laser-cut assembly.

Tactile media are important resources for people who are blind or have low vision (BLV) to access visual or graphical information as well as to develop tactile acuity. In this paper we present findings from a social learning design that was guided by the question, "What are the factors or issues that impact the tactile media consumption and production practices of BLV and other invested stakeholders?" From data collected during three Tactile Arts and Graphics Symposia-that gathered (N=64) BLV and sighted practitioners, researchers, and educators invested making information and experiences accessible through touch-we identified 34 issues that fall under four problems of practice, that impact tactile media practices. Based on these four problems of practice (and 34 underlying issues), we share recommendations to support the development of socio-technical systems that improve tactile access to information and full inclusion of people who are BLV in tactile media consumption and production.

We introduce shape-changing swarm robots. A swarm of self-transformable robots can both individually and collectively change their configuration to display information, actuate objects, act as tangible controllers, visualize data, and provide physical affordances. ShapeBots is a concept prototype of shape-changing swarm robots. Each robot can change its shape by leveraging small linear actuators that are thin (2.5 cm) and highly extendable (up to 20cm) in both horizontal and vertical directions. The modular design of each actuator enables various shapes and geometries of self-transformation. We illustrate potential application scenarios and discuss how this type of interface opens up possibilities for the future of ubiquitous and distributed shape-changing interfaces.

This article establishes and addresses opportunities for ethics integration into Machine-learning (ML) courses. Following a survey of the history of computing ethics and the current need for ethical consideration within ML, we consider the current state of ML ethics education via an exploratory analysis of course syllabi in computing programs. The results reveal that though ethics is part of the overall educational landscape in these programs, it is not frequently a part of core technical ML courses. To help address this gap, we offer a preliminary framework, developed via a systematic literature review, of relevant ethics questions that should be addressed within an ML project. A pilot study with 85 students confirms that this framework helped them identify and articulate key ethical considerations within their ML projects. Building from this work, we also provide three example ML course modules that bring ethical thinking directly into learning core ML content. Collectively, this research demonstrates: (1) the need for ethics to be taught as integrated within ML coursework, (2) a structured set of questions useful for identifying and addressing potential issues within an ML project, and (3) novel course models that provide examples for how to practically teach ML ethics without sacrificing core course content. An additional by-product of this research is the collection and integration of recent publications in the emerging field of ML ethics education.

Our interviews with people who have visual impairments show clothes shopping is an important activity in their lives. Unfortunately, clothes shopping web sites remain largely inaccessible. We propose design recommendations to address online accessibility issues reported by visually impaired study participants and an implementation, which we call BrowseWithMe, to address these issues. BrowseWithMe employs artificial intelligence to automatically convert a product web page into a structured representation that enables a user to interactively ask the BrowseWithMe system what the user wants to learn about a product (e.g., What is the price? Can I see a magnified image of the pants?). This enables people to be active solicitors of the specific information they are seeking rather than passive listeners of unparsed information. Experiments demonstrate BrowseWithMe can make online clothes shopping more accessible and produce accurate image descriptions.

Devising strategies to engage the public in discussions around the design and development of technology is critical to building a future that works for everyone. This paper presents a novel case study, an immersive theater experience, "Quantified Self," that combines aspects of design fiction and user enactments to construct a public engagement opportunity about technology ethics. Our audience supplied their social data (Facebook, Twitter...) and received a personalized experience where they interacted with a narrative and technology exhibits. We used a design model targeting goals of engagement, education, and discussion. Here we overview the design and production of Quantified Self and report on the results (240 participants over 6 performances) and findings from audience surveys (n=179/240) and cast/crew interviews (n=15/22). We found our approach attracted a wide audience interested in different elements of the show. Affordances and challenges of our model are discussed in detail.

One contributing factor to how people choose to use technology is their perceptions of associated risk. In order to explore this influence, we adapted a survey instrument from risk perception literature to assess mental models of users and technologists around risks of emerging, data-driven technologies (e.g., identity theft, personalized filter bubbles). We surveyed 175 individuals for comparative and individual assessments of risk, including characterizations using psychological factors. We report our observations around group differences (e.g., expert versus non-expert) in how people assess risk, and what factors may structure their conceptions of technological harm. Our findings suggest that technologists see these risks as posing a bigger threat to society than do non-experts. Moreover, across groups, participants did not see technological risks as voluntarily assumed. Differences in how people characterize risk have implications for the future of design, decision-making, and public communications, which we discuss through a lens we call risk-sensitive design.

We explore a new approach to programming swarm user interfaces (Swarm UI) by leveraging direct physical manipulation. Existing Swarm UI applications are written using a robot programming framework: users work on a computer screen and think in terms of low-level controls. In contrast, our approach allows programmers to work in physical space by directly manipulating objects and think in terms of high-level interface design. Inspired by current UI programming practices, we introduce a four-step workflow-create elements, abstract attributes, specify behaviors, and propagate changes-for Swarm UI programming. We propose a set of direct physical manipulation techniques to support each step in this workflow. To demonstrate these concepts, we developed Reactile, a Swarm UI programming environment that actuates a swarm of small magnets and displays spatial information of program states using a DLP projector. Two user studies-an in-class survey with 148 students and a lab interview with eight participants-confirm that our approach is intuitive and understandable for programming Swarm UIs.

We explore web programming as a new paradigm for programmatic 3D modeling. Most existing approaches subscribe to the imperative programming paradigm. While useful, there exists a gulf of evaluation between procedural steps and the intended structure. We present CraftML, a language providing a declarative syntax where the code is the structure. CraftML offers a rich set of programming features familiar to web developers of all skill levels, such as tags, hyperlinks, document object model, cascade style sheet, JQuery, string interpolation, template engine, data injection, and scalable vector graphics. We develop an online IDE to support CraftML development, with features such as live preview, search, module import, and parameterization. Using examples and case studies, we demonstrate that CraftML offers a low floor for beginners to make simple designs, a high ceiling for experts to build complex computational models, and wide walls to support many application domains such as education, data physicalization, tactile graphics, assistive devices, and mechanical components.

Our paper offers several novel activities for teaching ethics in the context of a computer science (CS) class. Rather than approaches that teach ethics as an isolated course, we outline and discuss multiple ethics education interventions meant to work in the context of an existing technical course. We piloted these activities in an Human Centered Computing course and found strong engagement and interest from our students in ethics topics without sacrificing core course material. Using a pre/post survey and examples from student assignments, we evaluate the impact of these interventions and discuss their relevance to other CS courses. We further make suggestions for embedding ethics in other CS education contexts.

This paper presents Quantified Self: Immersive Data and Theater Experience (QSelf) as a case study in collaborative and interdisciplinary learning and toward a project-based education model that promotes technical art projects. 22 students from several departments engaged in a semester-long effort to produce an immersive theater show centered on ethical uses of personal data, a show that drew more than 240 people over 6 performances. The project was housed out of the computer science department and involved multiple computer science undergraduate and graduate students who had the chance to work with students from the department of theater and dance. By analyzing the technical artifacts students created and post-interviews, we found this project created a novel and productive space for computer science students to gain applied experience and learn about the social impacts of their work while the arts students gained a fluency and understanding around the technical issues presented.

This paper introduces Dynamic 3D Printing, a fast and reconstructable shape formation system. Dynamic 3D Printing can assemble an arbitrary three-dimensional shape from a large number of small physical elements. Also, it can disassemble the shape back to elements and reconstruct a new shape. Dynamic 3D Printing combines the capabilities of 3D printers and shape displays: Like conventional 3D printing, it can generate arbitrary and graspable three-dimensional shapes, while allowing shapes to be rapidly formed and reformed as in a shape display. To demonstrate the idea, we describe the design and implementation of Dynablock, a working prototype of a dynamic 3D printer. Dynablock can form a three-dimensional shape in seconds by assembling 3,000 9 mm blocks, leveraging a 24 x 16 pin-based shape display as a parallel assembler. Dynamic 3D printing is a step toward achieving our long-term vision in which 3D printing becomes an interactive medium, rather than the means for fabrication that it is today. In this paper, we explore possibilities for this vision by illustrating application scenarios that are difficult to achieve with conventional 3D printing or shape display systems.

The growing accessibility of 3D printing to everyday users has led to the rapid adoption, sharing of 3D models on sites such as Thingiverse.com, and visions of a future in which customization is a norm and 3D printing can solve a variety of real-world problems. However, in practice, creating models is difficult and many end users simply print models created by others. In this paper, we explore a specific area of model design that is a challenge for end users' measurement. When a model must conform to a specific real world goal once printed, it is important that that goal is precisely specified. We demonstrate that measurement errors are a significant (yet often overlooked) challenge for end users through a systematic study of the sources and types of measurement errors. We argue for a new design principle--accommodating measurement error--that designers, as well as novice modelers, should to use at design time. We offer two strategies--buffer insertion and replacement of minimal parts--to help designers, as well as novice modelers, to build models that are robust to measurement error. We argue that these strategies can reduce the need for and costs of iteration and demonstrate their use in a series of printed objects.

For people with visual impairments, tactile graphics are an important means to learn and explore information. However, raised line tactile graphics created with traditional materials such as embossing are static. While available refreshable displays can dynamically change the content, they are still too expensive for many users, and are limited in size. These factors limit wide-spread adoption and the representation of large graphics or data sets. In this paper, we present FluxMaker, an inexpensive scalable system that renders dynamic information on top of static tactile graphics with movable tactile markers. These dynamic tactile markers can be easily reconfigured and used to annotate static raised line tactile graphics, including maps, graphs, and diagrams. We developed a hardware prototype that actuates magnetic tactile markers driven by low-cost and scalable electromagnetic coil arrays, which can be fabricated with standard printed circuit board manufacturing. We evaluate our prototype with six participants with visual impairments and found positive results across four application areas: location finding or navigating on tactile maps, data analysis, and physicalization, feature identification for tactile graphics, and drawing support. The user study confirms advantages in application domains such as education and data exploration.

Common appliances have shifted toward flat interface panels, making them inaccessible to blind people. Although blind people can label appliances with Braille stickers, doing so generally requires sighted assistance to identify the original functions and apply the labels. We introduce Facade - a crowdsourced fabrication pipeline to help blind people independently make physical interfaces accessible by adding a 3D printed augmentation of tactile buttons overlaying the original panel. Facade users capture a photo of the appliance with a readily available fiducial marker (a dollar bill) for recovering size information. This image is sent to multiple crowd workers, who work in parallel to quickly label and describe elements of the interface. Facade then generates a 3D model for a layer of tactile and pressable buttons that fits over the original controls. Finally, a home 3D printer or commercial service fabricates the layer, which is then aligned and attached to the interface by the blind person. We demonstrate the viability of Facade in a study with 11 blind participants.

Recent advances in 3D printing allowed end users to easily utilize desktop 3D printers. However, these printers mainly consume molten plastic, due to limited materiality. While high interests on creating interactive 3D objects with electronics are ubiquitous, commercial 3D printers have its own boundaries of not integrating electronics. On the other hand, paper as flexible medium has shown its possibility to be a channel to host interactive objects, enabling interaction triggered by user. In this paper, we introduce a process of integrating paper circuit into a 3D printed object, combining two universal fabrication techniques, 3D printing and paper crafting: 1) design a hollow 3D model, 2) construct circuits on the 2D planar cutout of this figure, 3) pause 3D printing to insert folded circuits in, 4) resume to continue printing. We empower makers and tinkerers to employ pervasive technology to build interactive 3D objects without hacking machine, obtaining professional utilities, or buying expensive materials.

Digital keypads have proliferated on common appliances, from microwaves and refrigerators to printers and remote controls. For blind people, such interfaces are inaccessible. We conducted a formative study with 6 blind people which demonstrated a need for custom designs for tactile labels without dependence on sighted assistance. To address this need, we introduce Facade - a crowdsourced fabrication pipeline to make physical interfaces accessible by adding a 3D printed augmentation of tactile buttons overlaying the original panel. Blind users capture a photo of an inaccessible interface with a standard marker for absolute measurements using perspective transformation. Then this image is sent to multiple crowd workers, who work in parallel to quickly label and describe elements of the interface. These labels are then used to generate 3D models for a layer of tactile and pressable buttons that fits over the original controls. Users can customize the shape and labels of the buttons using a web interface. Finally, a consumer-grade 3D printer fabricates the layer, which is then attached to the interface using adhesives. Such fabricated overlay is an inexpensive ($10) and more general solution to making physical interfaces accessible.

In this paper, we propose a prototype of music visualization system that captures and records the music component into digital data form, and then displays the data in visual form for children with hearing impairments. The analog sound data of music played physically is scaled into a binary matrix and scalar values that is then used as data structures for transcribing the output. We designed a system that detects tune and speed from a physical violin, and demonstrated three tangible music visualizations that children see in their daily lives, employing a flowerpot, plants and a picture of frame. We describe how the data captured from physical musical instruments can be seen through these objects, and suggest future possibilities for interactive sound visualization in music education for children with hearing impairments.

The design of 3D printable accessible tactile pictures (3DP-ATPs) for young children with visual impairments has the potential to greatly increase the supply of tactile materials that can be used to support emergent literacy skill development. Many caregivers and stakeholders invested in supporting young children with visual impairments have shown interest in using 3D printing to make accessible tactile materials. Unfortunately, the task of designing and producing 3DP-ATPs is far more complex than simply learning to use personal fabrication tools. This paper presents formative research conducted to investigate how six caregiver stakeholder-groups, with diverse skillsets and domain interests, attempt to create purposeful 3DP-ATPs with amateur-focused 3D modeling programs. We expose the experiences of these stakeholder groups as they attempt to design 3DP-ATG for the first time. We discuss how the participant groups practically and conceptually approach the task and focus their design work. Each group demonstrated different combinations of skillsets. In turn, we identify the common activities required of the design task as well how different participants are well suited and motivated to preform those activities. This study suggests that the emerging community of amateur 3DP-ATP designers may benefit from an online creativity support tool to help offset the challenges of designing purposeful 3DP-ATPs that are designed to meet individual children with VI's emergent literacy needs.

Many children's books contain movable pictures with elements that can be physically opened, closed, pushed, pulled, spun, flipped, or swung. But these tangible, interactive reading experiences are inaccessible to children with visual impairments. This paper presents a set of 3D-printable models designed as building blocks for creating movable tactile pictures that can be touched, moved, and understood by children with visual impairments. Examples of these models are canvases, connectors, hinges, spinners, sliders, lifts, walls, and cutouts. They can be used to compose movable tactile pictures to convey a range of spatial concepts, such as in/out, up/down, and high/low. The design and development of these models were informed by three formative studies including 1) a survey on popular moving mechanisms in children's books and 3D-printed parts to implement them, 2) two workshops on the process creating movable tactile pictures by hand (e.g., Lego, Play-Doh), and 3) creation of wood-based prototypes and an informal testing on sighted preschoolers. Also, we propose a design language based on XML and CSS for specifying the content and structure of a movable tactile picture. Given a specification, our system can generate a 3D-printable model. We evaluate our approach by 1) transcribing six children's books, and 2) conducting six interviews on domain experts including four teachers for the visually impaired, one blind adult, two publishers at the National Braille Press, a renowned tactile artist, and a librarian.

Mobile user interface design patterns have been widely used across different mobile platforms. UI design patterns have evolved and changed significantly as new trends emerge and fade at different times. This paper presents a data-mining approach to analyzing design pattern changes in Android apps. Over a period of 18 months, we tracked 24,436 apps and collected their versions. In total, our sample consists of 56,349 unique app versions, more than 5 million source files, and more than 25 million UI elements. We developed a dedicated infrastructure based on modern big data technologies to support our differential analyses regarding design pattern changes. Some highlights of our findings include a) some apps would switch to a design pattern even after it was deprecated, b) the adoption rate of newly introduced design patterns (e.g., Fragment) is relatively low, c) some apps would update their listing details to reflect changes in design patterns.

Traditionally, the inverted (or flipped) classroom has students complete traditional, passive learning tasks (e.g., watching lectures) while at home and uses class time to actualize what is learned through labs, discussions, and exercises. In this paper, we present an instructional model for teaching computer science (CS) that compounds features of the flipped classroom with components of peer instruction and formative assessment. Outside of class, in lieu of a lecture, students worked collaboratively on learning challenges that introduced content through a series of hands-on exercises. During class time, we used hack-a-thons to create an active classroom environment to promote peer coding and cultivate the growth of relevant real-world technical skills. Class work was digitally synced to Google Drive in real-time to allow instructors the opportunity to customize on-the-spot feedback. Further, we used journals as a formative assessment measure to synthesize student interests and opinions into our continued design of the class. In this paper, we describe our pedagogical model and discuss the results and lessons learned from the class using mined data from Google Drive and student journal responses.

3D Printing has shown a great potential to print tactile picture books, in order to cultivate emergent literacy for children with visual impairments. However, currently available 3D design tools are hard to learn, resulting in children to be excluded from the participatory design of tactile pictures. Also, existing 3D design software lacks of functionality to incorporate mobility and rich textures, which is critical aspect of the effective tactile picture. In this paper, we review our formative studies, presenting a hands-on design process for children to empower their own creativities into 3D tactile pictures design, and to engage them to bring other materials to enhance touch experiences.

Young children with visual impairments greatly benefit from tactile graphics (illustrations, images, puzzles, objects) during their learning processes. In this paper we present insight about using a 3D printed tactile picture book as a design probe. This has allowed us to identify and engage stakeholders in our research on improving the technical and human processes required for creating 3D printed tactile pictures, and cultivate a community of practice around these processes. We also contribute insight about how our inperson and digital methods of interacting with teachers, parents, and other professionals dedicated to supporting children with visual impairments contributes to research practices.

Biometric authentication relies on an individual's inner characteristics and traits. We propose an active authentication system on a mobile device that relies on two biometric modalities: 3D gestures and face recognition. The novelty of our approach is to combine 3D gesture and face recognition in a nonintrusive and unconstrained environment; the active authentication system is running in the background while the user is performing his/her main task.

Users make lasting judgments about a website's appeal within a split second of seeing it for the first time. This first impression is influential enough to later affect their opinions of a site's usability and trustworthiness. In this paper, we demonstrate a means to predict the initial impression of aesthetics based on perceptual models of a website's colorfulness and visual complexity. In an online study, we collected ratings of colorfulness, visual complexity, and visual appeal of a set of 450 websites from 548 volunteers. Based on these data, we developed computational models that accurately measure the perceived visual complexity and colorfulness of website screenshots. In combination with demographic variables such as a user's education level and age, these models explain approximately half of the variance in the ratings of aesthetic appeal given after viewing a website for 500ms only.

In this paper, we present a promising approach to systematically testing graphical user interfaces (GUI) in a platform independent manner. Our framework uses standard computer vision techniques through a python-based scripting language (Sikuli script) to identify key graphical elements in the screen and automatically interact with these elements by simulating keypresses and pointer clicks. The sequence of inputs and outputs resulting from the interaction is analyzed using grammatical inference techniques that can infer the likely internal states and transitions of the GUI based on the observations. Our framework handles a wide variety of user interfaces ranging from traditional pull down menus to interfaces built for mobile platforms such as Android and iOS. Furthermore, the automaton inferred by our approach can be used to check for potentially harmful patterns in the interface's internal state machine such as design inconsistencies (eg,. a keypress does not have the intended effect) and mode confusion that can make the interface hard to use. We describe an implementation of the framework and demonstrate its working on a variety of interfaces including the user-interface of a safety critical insulin infusion pump that is commonly used by type-1 diabetic patients.

The increasing number of images available online has created a growing need for efficient ways to search for relevant content. Text-based query search is the most common approach to retrieve images from the Web. In this approach, the similarity between the input query and the metadata of images is used to find relevant information. However, as the amount of available images grows, the number of relevant images also increases, all of them sharing very similar metadata but differing in other visual characteristics. This paper studies the influence of visual aesthetic quality in search results as a complementary attribute to relevance. By considering aesthetics, a new ranking parameter is introduced aimed at improving the quality at the top ranks when large amounts of relevant results exist. Two strategies for aesthetic rating inference are proposed: one based on visual content, another based on the analysis of user comments to detect opinions about the quality of images. The results of a user study with $58$ participants show that the comment-based aesthetic predictor outperforms the visual content-based strategy, and reveals that aesthetic-aware rankings are preferred by users searching for photographs on the Web.

Pixel-based methods are emerging as a new and promising way to develop new interaction techniques on top of existing user interfaces. However, in order to maintain platform independence, other available low-level information about GUI widgets, such as accessibility metadata, was neglected intentionally. In this paper, we present a hybrid framework, PAX, which associates the visual representation of user interfaces (i.e. the pixels) and their internal hierarchical metadata (i.e. the content, role, and value). We identify challenges to building such a framework. We also develop and evaluate two new algorithms for detecting text at arbitrary places on the screen, and for segmenting a text image into individual word blobs. Finally, we validate our framework in implementations of three applications. We enhance an existing pixel-based system, Sikuli Script, and preserve the readability of its script code at the same time. Further, we create two novel applications, Screen Search and Screen Copy, to demonstrate how PAX can be applied to development of desktop-level interactive systems.

Contextual help is effective for learning how to use GUIs by showing instructions and highlights on the actual interface rather than in a separate viewer. However, end-users and third-party tech support typically cannot create contextual help to assist other users because it requires programming skill and source code access. We present a creation tool for contextual help that allows users to apply common computer skills-taking screenshots and writing simple scripts. We perform pixel analysis on screenshots to make this tool applicable to a wide range of applications and platforms without source code access. We evaluated the tool's usability with three groups of participants: developers, in-structors, and tech support. We further validated the applicability of our tool with 60 real tasks supported by the tech support of a university campus.

The multimedia information retrieval community has dedicated extensive research effort to the problem of content-based image retrieval (CBIR). However, these systems find their main limitation in the difficulty of creating pictorial queries. As a result, few systems offer the option of querying by visual examples, and rely on automatic concept detection and tagging techniques to provide support for searching visual content using textual queries.

Testing a GUI's visual behavior typically requires human testers to interact with the GUI and to observe whether the expected results of interaction are presented. This paper presents a new approach to GUI testing using computer vision for testers to automate their tasks. Testers can write a visual test script that uses images to specify which GUI components to interact with and what visual feedback to be observed. Testers can also generate visual test scripts by demonstration. By recording both input events and screen images, it is possible to extract the images of components interacted with and the visual feedback seen by the demonstrator, and generate a visual test script automatically. We show that a variety of GUI behavior can be tested using this approach. Also, we show how this approach can facilitate good testing practices such as unit testing, regression testing, and test-driven development.

The lack of access to visual information like text labels, icons, and colors can cause frustration and decrease independence for blind people. Current access technology uses automatic approaches to address some problems in this space, but the technology is error-prone, limited in scope, and quite expensive. In this paper, we introduce VizWiz, a talking application for mobile phones that offers a new alternative to answering visual questions in nearly real-time - asking multiple people on the web. To support answering questions quickly, we introduce a general approach for intelligently recruiting human workers in advance called quikTurkit so that workers are available when new questions arrive. A field deployment with 11 blind participants illustrates that blind people can effectively use VizWiz to cheaply answer questions in their everyday lives, highlighting issues that automatic approaches will need to address to be useful. Finally, we illustrate the potential of using VizWiz as part of the participatory design of advanced tools by using it to build and evaluate VizWiz::LocateIt, an interactive mobile tool that helps blind people solve general visual search problems.

Object localization and recognition are important problems in computer vision. However, in many applications, exhaustive search over all object models and image locations is computationally prohibitive. While several methods have been proposed to make either recognition or localization more efficient, few have dealt with both tasks simultaneously. This paper proposes an efficient method for concurrent object localization and recognition based on a data-dependent multi-class branch-and-bound formalism. Existing bag-of-features recognition techniques which can be expressed as weighted combinations of feature counts can be readily adapted to our method. We present experimental results that demonstrate the merit of our algorithm in terms of recognition accuracy, localization accuracy, and speed, compared to baseline approaches including exhaustive search, implicit-shape model (ISM), and efficient sub-window search (ESS). Moreover, we develop two extensions to consider non-rectangular bounding regions-composite boxes and polygons-and demonstrate their ability to achieve higher recognition scores compared to traditional rectangular bounding boxes.

We describe a mixed-modality method to index and search software documentation in three ways: plain text, OCR text of embedded figures, and visual features of these figures. Using a corpus of 102 computer books with a total of 62,943 pages and 75,800 figures, we empirically demonstrate that our method achieves better precision/recall than do alternatives based on single modalities.

We present Sikuli, a visual approach to search and automation of graphical user interfaces using screenshots. Sikuli allows users to take a screenshot of a GUI element (such as a toolbar button, icon, or dialog box) and query a help system using the screenshot instead of the element's name. Sikuli also provides a visual scripting API for automating GUI interactions, using screenshot patterns to direct mouse and keyboard events. We report a web-based user study showing that searching by screenshot is easy to learn and faster to specify than keywords. We also demonstrate several automation tasks suitable for visual scripting, such as map navigation and bus tracking, and show how visual scripting can improve interactive help systems previously proposed in the literature.

Dynamic visual category learning calls for efficient adaptation as new training images become available or new categories are defined, existing training images or categories become modified or obsolete, or when categories are divided into subcategories or merged together. We develop novel methods for efficient incremental learning of SVM-based visual category classifiers to handle such dynamic tasks. Our method exploits previous classifier estimates to more efficiently learn the optimal parameters for the current set of training images and categories. We show empirically that for dynamic visual category tasks, our incremental learning methods are significantly faster than batch retraining.

This paper introduces multimodal question answering, a new interface for community-based question answering services. By offering users an extra modality---photos---in addition to the text modality to formulate queries, multimodal question answering overcomes the limitations of text-only input methods when the users ask questions regarding visually distinctive objects. Such interface is especially useful when users become curious about an interesting object in the environment and want to know about it---simply by taking a photo and asking a question in a situated (from a mobile device) and intuitive (without describing the object in words) manner. We propose a system architecture for multimodal question answering, describe an algorithm for searching the database, and report on the findings of two prototype studies.

Photo-based question answering is a useful way of finding information about physical objects. Current question answering (QA) systems are text-based and can be difficult to use when a question involves an object with distinct visual features. A photo-based QA system allows direct use of a photo to refer to the object. We develop a three-layer system architecture for photo-based QA that brings together recent technical achievements in question answering and image matching. The first, template-based QA layer matches a query photo to online images and extracts structured data from multimedia databases to answer questions about the photo. To simplify image matching, it exploits the question text to filter images based on categories and keywords. The second, information retrieval QA layer searches an internal repository of resolved photo-based questions to retrieve relevant answers. The third, human-computation QA layer leverages community experts to handle the most difficult cases. A series of experiments performed on a pilot dataset of 30,000 images of books, movie DVD covers, grocery items, and landmarks demonstrate the technical feasibility of this architecture. We present three prototypes to show how photo-based QA can be built into an online album, a text-based QA, and a mobile application.

Histogram pyramid representations computed from a vocabulary tree of visual words have proven valuable for a range of image indexing and recognition tasks; however, they have only used a single, fixed partition of feature space. We present a new efficient algorithm to incrementally compute set-of-trees (forest) vocabulary representations, and show that they improve recognition and indexing performance in methods which use histogram pyramids. Our algorithm incrementally adapts a vocabulary forest with an Inverted file system at the leaf nodes and automatically keeps existing histogram pyramid database entries up-to-date in a forward filesystem. It is possible not only to apply vocabulary tree indexing algorithms directly, but also to compute pyramid match kernel values efficiently. On dynamic recognition tasks where categories or objects under consideration may change over time, we show that adaptive vocabularies offer significant performance advantages in comparison to a single, fixed vocabulary.

In this paper, we describe an intelligent user interface designed for camera phones to allow mobile users to specify the object of interest in the scene simply by taking two pictures: one with the object and one without the object. By comparing these two images, the system can reliably extract the visual appearance of the object, which can be useful to a wide-range of applications such as content-based image retrieval and object recognition.