Online fashion shopping is booming, yet high return rates persist as actual size and fit do not meet shoppers’ expectations. Virtual Try-On (VTON), rendering garments on individuals’ body images, promises to reduce such issues. To understand VTON’s impact on shopping behaviors and experiences, we conducted user study with 24 participants where they were asked to explore and purchase clothing online and then shared their thoughts on satisfaction, similarity, and return intentions after wearing items. Results show that VTON reduced exploration time and product detail views, while enabling clearer expectations of fit before delivery. Importantly, participants often used VTON for final verification, while some sought to discover new styles, suggesting VTON should provide adaptive support according to users’ tendencies. Participants also emphasized fit accuracy, highlighting the need for technical improvements and reliability cues such as confidence scores. Building on these findings, we suggest design implications for integrating VTON into e-commerce.

The rise of Large Language Models (LLMs) has ushered in a wave of conversational search engines that allow people to engage in dialogues with LLM-infused chatbots to seek information. As people tend to infer personalities from digital social interactions, and given that personality cues have been shown to affect credibility, these perceptions of chatbot design may shape how users assess the credibility of information in conversational search. In this study, we conducted a controlled online study with 190 participants who assessed conversational search results with chatbots designed to exhibit different levels of personality traits. We found that in conversational search, personality can affect perceptions of credibility. Specifically, perceived conscientiousness and agreeableness of a chatbot can increase credibility, while perceived extraversion and neuroticism can decrease the credibility of the information. This research contributes to our understanding of how conversational interfaces and their personality and persona designs can impact credibility. We also provide design implications for conversational search interfaces based on our findings.

Interacting with close-range objects in Virtual Reality (VR) is often prompted by visual cues, making it hard for visually impaired people to perceive their location and interact with them. To study how to enable blind users to locate and interact with close virtual objects, we adapted the arcade Speed-of-Light game as a blind-accessible VR application. We implemented three techniques: 1) Speech Feedback (e.g., “Top Right”), 2) Sonification, and 3) 2D Grid Position (e.g., “A3” for column and row); and conducted a user study with 15 blind participants aiming to provide insights into the design of non-visual techniques that convey information about targets at arm-reach. Speech Feedback was the most intuitive overall but verbose and the least flexible, while 2D Grid Position was found straightforward for regular spreadsheet users. Results also showed greater difficulty with Sonification, although it was valued by few participants who appreciated the challenge.

To improve the accessibility of visual figures, auto-generation of text description of individual images has been studied. However, it cannot be directly applied to comics as the descriptions can be redundant as similar scenes appear in a row. To address this issue, we propose generating the descriptions per group of related images and demonstrate how an dense captioning technique for videos can be utilized for this purpose and ways to improve its performance. To assess the effectiveness of our approach and to identify factors affecting the quality of text descriptions of comics, we conducted a preliminary study with 3 sighted evaluators and a main user study with 12 participants with visual impairments. The results show that text descriptions generated per group of images are perceived to be better than those generated per image in terms of accuracy, clarity, understandability, length, informativeness and preference for sighted groups, when annotator is human. In the same conditions, when the annotator is AI, it exhibited better performance in terms of length. Also, people with visual impairments prefer group descriptions because of conciseness, smooth connectivity of sentences, and non-repetitive features. Based on the findings, we provide design recommendations for generating accessible comic descriptions at a scale for blind users.

Large-scale 3D point clouds are often used as training data for 3D semantic segmentation, but the labor-intensive nature of the annotation process challenges the acquisition of sufficient labeled data. Meanwhile, there has been limited research on introducing novice annotators to acquire the labeled data by enhancing their annotation performance and user experience. Therefore, in this study, we explored solutions involving two dimensions: the presence of AI assistance and the number of classes visualized simultaneously in model’s segmentation results in HITL. We conducted a user study with 16 novice annotators who had no prior experience in 3D semantic segmentation, asking them to perform annotation tasks. The results revealed an interaction effect between the two dimensions on annotation accuracy and labeling efficiency. We also found that displaying multiple classes at once reduced the time taken for annotation. Moreover, visualizing multiple classes at once or the absence of AI assistance led to a greater increase in model accuracy compared to our baselines. The best user experience was observed when the visualization showed a single class at a time with AI assistance. Based on these findings, we discuss which environments can enhance novice annotators’ annotation performance and user experience in 3D semantic segmentation tasks within HITL contexts.

Podcasts present challenges in information retrieval due to their non-visual nature and extended length. To understand these challenges, we conducted interviews with 12 podcast users and identified difficulties in grasping the overall podcast content with metadata alone, highlighting the necessity of navigating to specific segments. Based on this finding, we propose a browsing method that utilizes Large Language Models (LLMs) and image generation models to segment podcast contents, integrating visual cues for supporting efficient navigation. To investigate how this new method differs from conventional approaches and to evaluate its effectiveness, we conducted another user study with 12 participants. The results revealed that keyword search is ineffective when dealing with unfamiliar or inaccurate keywords. Additionally, it requires thorough examination of the script to comprehend the overall content of each episode. On the other hand, segmenting the contents and labeling the topic for each segment facilitated was found to be helpful for understanding of the overall content, enabling easy navigation to desired topics. Furthermore, we found that providing an image enabled participants to easily distinguish one segment from another, which was preferred by participants. This multimodal browsing approach is expected to establish a foundational framework for the effective browsing and comprehension of audio content, extending its applicability beyond podcasts to various forms of audio files.

As virtual reality (VR) is typically designed in terms of visual experience, it poses major challenges for blind people to understand and interact with the environment. To address this, we propose a design space to explore how to augment objects and their behaviours in VR with a nonvisual audio representation. It intends to support designers in creating accessible experiences by explicitly considering alternative representations to visual feedback. To demonstrate its potential, we recruited 16 blind users and explored the design space under two scenarios in the context of boxing: understanding the location of objects (the opponent's defensive stance) and their movement (opponent's punches). We found that the design space enables the exploration of multiple engaging approaches for the auditory representation of virtual objects. Our findings depicted shared preferences but no one-size-fits-all solution, suggesting the need to understand the consequences of each design choice and their impact on the individual user experience.

Although there is a potential demand for customizing voices, most customization is limited to the visual appearance of a figure (e.g., avatars). To better understand the users’ need, we first conducted an online survey with 104 participants. Then we conducted a semi-structured interview with a prototype with 14 participants to identify design considerations for supporting voice customization. The results show that there is a desire for voice customization especially for non-face-to-face conversations with someone unfamiliar. In addition, the findings revealed that different voices are favored for different contexts from a better version of one’s own voice for improving delivery to a completely different voice for securing identity. As future work, we plan to extend this study by investigating voice synthesis techniques for end-users who wish to design their own voices for various contexts.

Webtoon is a type of digital comics read online where readers can leave comments to share their thoughts on the story. While it has experienced a surge in popularity internationally, people with visual impairments cannot enjoy webtoon with the lack of an accessible format. While traditional image description practices can be adopted, resulting descriptions cannot preserve webtoons’ unique values such as control over the reading pace and social engagement through comments. To improve the webtoon reading experience for BLV users, we propose Cocomix, an interactive webtoon reader that leverages comments into the design of novel webtoon interactions. Since comments can identify story highlights and provide additional context, we designed a system that provides 1) comments-based adaptive descriptions with selective access to details and 2) panel-anchored comments for easy access to relevant descriptive comments. Our evaluation (N=12) showed that Cocomix users could adapt the description for various needs and better utilize comments.

Providing directional guidance is important especially for exploring unfamiliar environments. However, most studies are limited to two-dimensional guidance when many interactions happen in 3D spaces. Moreover, visual feedback that is often used to communicate the 3D position of a particular object may not be available in situations when the target is occluded by other objects or located outside of one’s field of view, or due to visual overload or light conditions. Inspired by a prior finding that showed users’ tendency of scanning a 3D space in one direction at a time, we propose two-step nonvisual omnidirectional guidance feedback designs varying the searching order where the guidance for the vertical location of the target (the altitude) is offered to the users first, followed by the horizontal direction of the target (the azimuth angle) and visa versa. To investigate its effect, we conducted the user study with 12 blind-folded sighted participants. Findings suggest that our proposed two-step guidance outperforms the default condition with no order in terms of task completion time and travel distance, particularly when the guidance in the horizontal direction is presented first. We plan to extend this work to assist with finding a target in 3D spaces in a real-world environment.

This paper investigates how touchscreen exploration and verbal feedback can be used to support blind people to access visual artwork. We present two artwork exploration modalities. The first one, attribute-based exploration, extends prior work on touchscreen image accessibility, and provides fine-grained segmentation of artwork visual elements; when the user touches an element, the associated attributes are read. The second one, hierarchical exploration, is designed with domain experts and provides multi-level segmentation of the artwork; the user initially accesses a general description of the entire artwork and then explores a coarse segmentation of the visual elements with the corresponding high-level descriptions; once selected, coarse segments are subdivided into fine-grained ones, which the user can access for more detailed descriptions.

We present ReCog, a mobile app that enables blind users to recognize objects by training a deep network with their own photos of such objects. This functionality is useful to differentiate personal objects, which cannot be recognized with pre-trained recognizers and may lack distinguishing tactile features. To ensure that the objects are well-framed in the captured photos, ReCog integrates a camera-aiming guidance that tracks target objects and instructs the user through verbal and sonification feedback to appropriately frame them.

One of the reasons that people with visual impairments have difficulties in enjoying artworks is the limited number of accessible artworks. To enable people with visual impairments to explore and understand various artworks independently, we built a touchscreen-based mobile application as a prototype focusing on 2D paintings which plays object-level verbal descriptions upon users' touch. To confirm the needs and to collect initial feedback from potential users of such a system, we conducted an exploratory study with 8 participants with visual impairments using the prototype as a design probe. Overall, participants appreciated the prototype as they can learn artworks with less physical and time constraints while listening for details of contents as they explore each painting by touch.

Navigation assistive technologies aim to improve the mobility of blind or visually impaired people. In particular, turn-by-turn navigation assistants provide sequential instructions to enable autonomous guidance towards a destination. A problem frequently addressed in the literature is to obtain accurate position and orientation of the user during such guidance. An orthogonal challenge, often overlooked in the literature, is how precisely navigation instructions are followed by users. In particular, imprecisions in following rotation instructions lead to rotation errors that can significantly affect navigation. Indeed, a relatively small error during a turn is amplified by the following frontal movement and can lead the user towards incorrect or dangerous paths. In this contribution, we study rotation errors and their effect on turn-by-turn guidance for individuals with visual impairments. We analyze a dataset of indoor trajectories of 11 blind participants guided along three routes through a multi-story shopping mall using NavCog, a turn-by-turn smartphone navigation assistant. We find that participants extend rotations by 17º on average. The error is not proportional to the expected rotation; instead, it is accentuated for "slight turns" (22.5º-60º), while "ample turns" (60º-120º) are consistently approximated to 90º. We generalize our findings as design considerations for engineering navigation assistance in real-world scenarios.

For screenreader users who are blind or visually impaired (VI), today's mobile devices, while reasonably accessible, are not necessarily efficient. This inefficiency may be especially problematic for microinteractions, which are brief but high-frequency interactions that take only a few seconds for sighted users to complete (e.g., checking the weather or for new messages). One potential solution to support efficient non-visual microinteractions is on-body input, which appropriates the user's own body as the interaction medium. In this paper, we address two related research questions: How well are microinteractions currently supported for VI users' How should on-body interaction be designed to best support microinteractions for this user group? We conducted two studies: (1) an online survey to compare current microinteraction use between VI and sighted users (N=117); and (2) an in-person study where 12 VI screenreader users qualitatively evaluated a real-time on-body interaction system that provided three contrasting input designs. Our findings suggest that efficient microinteractions are not currently well-supported for VI users, at least using manual input, which highlights the need for new interaction approaches. On-body input offers this potential and the qualitative evaluation revealed tradeoffs with different on-body interaction techniques in terms of perceived efficiency, learnability, social acceptability, and ability to use on the go.

Navigating in unfamiliar environments is challenging for most people, especially for individuals with visual impairments. While many personal navigation tools have been proposed to enable in- dependent indoor navigation, they have insufficient accuracy (e.g., 5-10 m), do not provide semantic features about surroundings (e.g., doorways, shops, etc.), and may require specialized devices to function. Moreover, the deployment of many systems is often only evaluated in constrained scenarios, which may not precisely reflect the performance in the real world. Therefore, we have de- signed and implemented NavCog3, a smartphone-based indoor navigation assistant that has been evaluated in a 21,000 m2 shop- ping mall. In addition to turn-by-turn instructions, it provides in- formation on landmarks (e.g., tactile paving) and points of interests nearby. We first conducted a controlled study with 10 visually im- paired users to assess localization accuracy and the perceived use- fulness of semantic features. To understand the usability of the app in a real-world setting, we then conducted another study with 43 participants with visual impairments where they could freely nav- igate in the shopping mall using NavCog3. Our findings suggest that NavCog3 can open a new opportunity for users with visual im- pairments to independently find and visit large and complex places with confidence.

Skin-based biometrics rely on the distinctiveness of skin patterns across individuals for identification. In this paper, we investigate whether small image patches of the skin can be localized on a user's body, determining not “who?” instead “where?” Applying techniques from biometrics and computer vision, we introduce a hierarchical classifier that estimates a location from the image texture and refines the estimate with keypoint matching and geometric verification. To evaluate our approach, we collected 10,198 close-up images of 17 hand and wrist locations across 30 participants. Within-person algorithmic experiments demonstrate that an individual's own skin features can be used to localize their skin surface image patches with an F1score of 96.5%. As secondary analyses, we assess the effects of training set size and between-person classification. We close with a discussion of the strengths and limitations of our approach and evaluation methods as well as implications for future applications using a wearable camera to support touch-based, location-specific taps and gestures on the surface of the skin.

The HandSight project investigates how wearable micro-cameras can be used to augment a blind or visually impaired user--s sense of touch with computer vision. Our goal is to support an array of activities of daily living by sensing and feeding back non-tactile information (e.g., color, printed text, patterns) about an object as it is touched. In this poster paper, we provide an overview of the project, our current proof-of-concept prototype, and a summary of findings from finger-based text reading studies. As this is an early-stage project, we also enumerate current open questions.

For users with visual impairments, who do not necessarily need the visual display of a mobile device, non-visual on-body interaction (e.g., Imaginary Interfaces) could provide accessible input in a mobile context. Such interaction provides the potential advantages of an always-available input surface, and increased tactile and proprioceptive feedback compared to a smooth touchscreen. To investigate preferences for and design of accessible on-body interaction, we conducted a study with 12 visually impaired participants. Participants evaluated five locations for on-body input and compared on-phone to on-hand interaction with one versus two hands. Our findings show that the least preferred areas were the face/neck and the forearm, while locations on the hands were considered to be more discreet and natural. The findings also suggest that participants may prioritize social acceptability over ease of use and physical comfort when assessing the feasibility of input at different locations of the body. Finally, tradeoffs were seen in preferences for touchscreen versus on-body input, with on-body input considered useful for contexts where one hand is busy (e.g., holding a cane or dog leash). We provide implications for the design of accessible on-body input.

With the increasing popularity of mainstream wearable devices, it is critical to assess the accessibility implications of such technologies. For people with visual impairments, who do not always need the visual display of a mobile phone, alternative means of eyes-free wearable interaction are particularly appealing. To explore the potential impacts of such technology, we conducted two studies. The first was an online survey that included 114 participants with visual impairments and 101 sighted participants; we compare the two groups in terms of current device use. The second was an interview and design probe study with 10 participants with visual impairments. Our findings expand on past work to characterize a range of trends in smartphone use and accessibility issues therein. Participants with visual impairments also responded positively to two eyes-free wearable device scenarios: a wristband or ring and a glasses-based device. Discussions on projected use of these devices suggest that small, easily accessible, and discreet wearable input could positively impact the ability of people with visual impairments to access information on the go and to participate in certain social interactions.

While sighted users may learn to perform touchscreen gestures through observation (e.g., of other users or video tutorials), such mechanisms are inaccessible for users with visual impairments. As a result, learning to perform gestures can be challenging. We propose and evaluate two techniques to teach touchscreen gestures to users with visual impairments: (1) corrective verbal feedback using text-to-speech and automatic analysis of the user's drawn gesture; (2) gesture sonification to generate sound based on finger touches, creating an audio representation of a gesture. To refine and evaluate the techniques, we conducted two controlled lab studies. The first study, with 12 sighted participants, compared parameters for sonifying gestures in an eyes-free scenario and identified pitch + stereo panning as the best combination. In the second study, 6 blind and low-vision participants completed gesture replication tasks with the two feedback techniques. Subjective data and preliminary performance findings indicate that the techniques offer complementary advantages.

The vast majority of work on understanding and supporting the gesture creation process has focused on professional designers. In contrast, gesture customization by end users' - which may offer better memorability, efficiency and accessibility than pre-defined gestures - has received little attention. To understand the end-user gesture creation process, we conducted a study where 20 participants were asked to: (1) exhaustively create new gestures for an open-ended use case; (2) exhaustively create new gestures for 12 specific use cases; (3) judge the saliency of different touchscreen gesture features. Our findings showed that even when asked to create novel gestures, participants tended to focus on the familiar. Misconceptions about the gesture recognizer's abilities were also evident, and in some cases constrained the range of gestures that participants created. Finally, as a calibration point for future research, we used a simple gesture recognizer ($N) to analyze recognition accuracy of the participants' custom gesture sets: accuracy was 68-88% on average, depending on the amount of training and the customization scenario. We conclude with implications for the design of a mixed-initiative approach to support custom gesture creation.