Blind and Low-vision (BLV) people face challenges in their daily lives, especially regarding independent navigation in unfamiliar settings. Assistive robots have recently emerged as assistive tools for people with special needs. Drones, in particular, have unique advantages over ground robots. Our work aims at designing a drone assistant in addition to primary mobility aids (i.e., white cane) to improve spatial awareness and navigation in unfamiliar settings. Using a Research-through-Design method with 21 BLV users, we identified challenging scenarios and the need for two distinct operating modes (Explorer and Team). We then deployed a low-fidelity prototype in a Wizard-of-Oz study. Participants were enthusiastic about the assistive drone. The outcomes were reviewed by low vision specialists and drone experts to critically assess the strengths and limitations of this assistive tool. Based on these results, we provide recommendations for future design of assistive drones tailored to BLV users.

Drones have gained traction as a versatile form of assistive robots for Blind and Low Vision (BLV) people. Nonetheless, novel interaction techniques are required to enable BLV people to communicate with drones naturally. In this work, we built an LLM-powered assistive drone for BLV users. We leverage an LLM to translate high-level user goals to step-by-step instructions for the drone and to extract visual information from the images. Through a formative study with BLV users (N=9), we identified envisioned use cases and desired interaction modalities. Then, we took a participatory and iterative approach to build a prototype, incorporating feedback received from 3 BLV users, as well as 5 domain experts. Finally, we conducted a user study with an additional 6 BLV participants to evaluate the iterated prototype, and received positive feedback. This work is contributing to a growing body of research on harnessing the power of LLMs to build a more inclusive world.

International audience

Programming has become a highly sought-after skill in STEM-related studies and careers, but it has only reached a fraction of students with visual impairments. Therefore, there is a need to explore new methods for teaching and learning. This study aims to understand the potential of using drones to create an authentic learning environment to help students with visual impairments learn programming. Based on a month-long engagement with five students with visual impairments, we present insights on using drones to support programming education for students with visual impairments.

The practice of sound design involves creating and manipulating environmental sounds for music, films, or games. Recently, an increasing number of studies have adopted generative AI to assist in sound design co-creation. Most of these studies focus on the needs of novices, and less on the pragmatic needs of sound design practitioners. In this paper, we aim to understand how generative AI models might support sound designers in their practice. We designed two interactive generative AI models as Creative Support Tools (CSTs) and invited nine professional sound design practitioners to apply the CSTs in their practice. We conducted semi-structured interviews and reflected on the challenges and opportunities of using generative AI in mixed-initiative interfaces for sound design. We provide insights into sound designers’ expectations of generative AI and highlight opportunities to situate generative AI-based tools within the design process. Finally, we discuss design considerations for human-AI interaction researchers working with audio.

In this paper, we propose a data-driven approach to train a Generative Adversarial Network (GAN) conditioned on "soft-labels" distilled from the penultimate layer of an audio classifier trained on a target set of audio texture classes. We demonstrate that interpolation between such conditions or control vectors provide smooth morphing between the generated audio textures, and show similar or better audio texture morphing capability compared to the state-of-the-art methods. The proposed approach results in a well-organized latent space that generates novel audio outputs while remaining consistent with the semantics of the conditioning parameters. This is a step towards a general data-driven approach to designing generative audio models with customized controls capable of traversing out-of-distribution regions for novel sound synthesis.