Nikolas Martelaro

dblp:147/0782 · also Nik Martelaro · DBLP profile ↗
← Back
38ranked-venue papers
7as first author
24since 2021 · last 2026
0000-0002-1824-0243ORCID · verified

Domains — the database's venue-derived domains; a paper can count in several

Human-computer interaction and ubiquitous computing · 38 · 7 first-author · 24 since 2021Artificial intelligence and machine learning · 5 · 2 first-author
YearPublicationVenuePosition
2026 EcoAssist: Embedding Sustainability into AI-Assisted Frontend Development
abstract
Frontend code, replicated across millions of page views, consumes significant energy and contributes directly to digital emissions. Yet current AI coding assistants, such as GitHub Copilot and Amazon CodeWhisperer, emphasize developer speed and convenience, with energy impact not yet a primary focus. At the same time, existing energy-focused guidelines and metrics have seen limited adoption among practitioners, leaving a gap between research and everyday coding practice. To address this gap, we introduce EcoAssist, an energy-aware assistant integrated into an IDE that analyzes AI-generated frontend code, estimates its energy footprint, and proposes targeted optimizations. We evaluated EcoAssist through benchmarks of 500 websites and a controlled study with 20 developers. Results show that EcoAssist reduced per-website energy by 13–16% on average, increased developers’ awareness of energy use, and maintained developer productivity. This work demonstrates how energy considerations can be embedded directly into AI-assisted coding workflows, supporting developers as they engage with energy implications through actionable feedback.
André Barrocas, Nuno Nunes 0001, Valentina Nisi, Nikolas Martelaro
CHI4
2026 Prototyping Multimodal GenAI Real-Time Agents with Counterfactual Replays and Hybrid Wizard-of-Oz
abstract
Recent advancements in multimodal generative AI (GenAI) enable the creation of personal context-aware real-time agents that, for example, can augment user workflows by following their on-screen activities and providing contextual assistance. However, prototyping such experiences is challenging, especially when supporting people with domain-specific tasks using real-time inputs such as speech and screen recordings. While prototyping an LLM-based proactive support agent system, we found that existing prototyping and evaluation methods were insufficient to anticipate the nuanced situational complexity and contextual immediacy required. To overcome these challenges, we explored a novel user-centered prototyping approach that combines counterfactual video replay prompting and hybrid Wizard of Oz methods to iteratively design and refine agent behaviors. This paper discusses our prototyping experiences, highlighting successes and limitations, and offers a practical guide and an open-source toolkit for UX designers, HCI researchers, and AI toolmakers to build more user-centered and context-aware multimodal agents.
Frederic Gmeiner, Kenneth Holstein, Nikolas Martelaro
CHI3
2026 Visual Lyrics: Generating Animated Text for Music Lyric Videos with an Augmented Text Editor
abstract
Animated lyric videos transform song lyrics into dynamic visual experiences, offering a powerful medium for artistic expression and audience engagement. However, creating these videos is challenging, requiring expertise in audio, typography, graphic design, and animation, making it inaccessible to novices. To address this challenge, we introduce Visual Lyrics, a proof-of-concept system for generating animated lyric videos controlled with an augmented text editor interface. We examined existing lyric videos to distill a taxonomy and design guidelines, informing the design of Visual Lyrics. Our key insight is a multimodal music analysis pipeline based on the taxonomy and leveraging LLM’s strong natural language understanding and code generation capabilities to synthesize creative and semantically meaningful animations. We collected a dataset of over 300 code-driven creative text animations to serve as inspiration for our LLM-driven pipeline, which we open source. In a user study, Visual Lyrics enabled novices to easily create high-quality animated lyric videos with high ratings of enjoyment, inspiration, and exploration.
David Chuan-En Lin, Cuong Nguyen 0003, Hijung Shin, Nikolas Martelaro
IUI4
2025 Exploring the Potential of Metacognitive Support Agents for Human-AI Co-Creation
abstract
Despite the potential of generative AI (GenAI) design tools to enhance design processes, professionals often struggle to integrate AI into their workflows. Fundamental cognitive challenges include the need to specify all design criteria as distinct parameters upfront (intent formulation) and designers' reduced cognitive involvement in the design process due to cognitive offloading, which can lead to insufficient problem exploration, underspecification, and limited ability to evaluate outcomes. Motivated by these challenges, we envision novel metacognitive support agents that assist designers in working more reflectively with GenAI. To explore this vision, we conducted exploratory prototyping through a Wizard of Oz elicitation study with 20 mechanical designers probing multiple metacognitive support strategies. We found that agent-supported users created more feasible designs than non-supported users, with differing impacts between support strategies. Based on these findings, we discuss opportunities and tradeoffs of metacognitive support agents and considerations for future AI-based design tools.
Frederic Gmeiner, Kaitao Luo, Kenneth Holstein, Nikolas Martelaro
Conference on Designing Interactive Systems5
2025 At the Breaking Point: How Bus Operators Cope with Transit Technology Failures and What That Can Tell Us About the Integration of Future Innovations
abstract
This paper examines the cascading effects of technical failures in transit, focusing on the challenges faced by bus operators when communication, passenger-facing, and mechanical technologies fail.Through a diary study, we gather operator accounts of critical tools like radios, mobile data terminals (MDTs), payment systems, and ramps, alongside their failures.Issues like radio outages, GPS malfunctions, and broken fare systems lead to operational delays, safety risks, and increased stress.Triaging breakdowns becomes crucial to operations and drivers adapt by using personal phones in emergencies, highlighting gaps in system integration.As transit electrifies its fleets and considers a wider range of innovations, these failures offer key insights into the challenges ahead, emphasizing the need for robust, adaptable systems that ensure operational continuity and protect worker well-being amid rapid technological change.
Alice Xiaodi Tang, Hunter Akridge, Nikolas Martelaro, Sarah E. Fox
Conference on Designing Interactive Systems3
2025 Non-Emergency Notification Timing for Drivers Doing Non-Driving-Related Tasks in Autonomous Vehicles: An Interruptibility Study
Hongyu Howie Wang, Jiya Gupta, Nikolas Martelaro
AutomotiveUI3
2025 Intent Tagging: Exploring Micro-Prompting Interactions for Supporting Granular Human-GenAI Co-Creation Workflows
abstract
Despite Generative AI (GenAI) systems' potential for enhancing content creation, users often struggle to effectively integrate GenAI into their creative workflows. Core challenges include misalignment of AI-generated content with user intentions (intent elicitation and alignment), user uncertainty around how to best communicate their intents to the AI system (prompt formulation), and insufficient flexibility of AI systems to support diverse creative workflows (workflow flexibility). Motivated by these challenges, we created IntentTagger: a system for slide creation based on the notion of Intent Tags - small, atomic conceptual units that encapsulate user intent - for exploring granular and non-linear micro-prompting interactions for Human-GenAI co-creation workflows. Our user study with 12 participants provides insights into the value of flexibly expressing intent across varying levels of ambiguity, meta-intent elicitation, and the benefits and challenges of intent tag-driven workflows. We conclude by discussing the broader implications of our findings and design considerations for GenAI-supported content creation workflows.
Frederic Gmeiner, Nicolai Marquardt, Michael Bentley, Hugo Romat, Michel Pahud, Asta Roseway, Nikolas Martelaro, Kenneth Holstein, Ken Hinckley, Nathalie Henry Riche
CHI8
2025 BioSpark: Beyond Analogical Inspiration to LLM-augmented Transfer
abstract
We present BioSpark, a system for analogical innovation designed to act as a creativity partner in reducing the cognitive effort in finding, mapping, and creatively adapting diverse inspirations. While prior approaches have focused on initial stages of finding inspirations, BioSpark uses LLMs embedded in a familiar, visual, Pinterest-like interface to go beyond inspiration to supporting users in identifying the key solution mechanisms, transferring them to the problem domain, considering tradeoffs, and elaborating on details and characteristics. To accomplish this BioSpark introduces several novel contributions, including a tree-of-life enabled approach for generating relevant and diverse inspirations, as well as AI-powered cards including 'Sparks' for analogical transfer; 'Trade-offs' for considering pros and cons; and 'Q&A' for deeper elaboration. We evaluated BioSpark through workshops with professional designers and a controlled user study, finding that using BioSpark led to a greater number of generated ideas; those ideas being rated higher in creative quality; and more diversity in terms of biological inspirations used than a control condition. Our results suggest new avenues for creativity support tools embedding AI in familiar interaction paradigms for designer workflows.
Hyeonsu B. Kang, David Chuan-En Lin, Yan-Ying Chen, Matthew K. Hong, Nikolas Martelaro, Aniket Kittur
CHI5
2025 Inkspire: Supporting Design Exploration with Generative AI through Analogical Sketching
abstract
With recent advancements in the capabilities of Text-to-Image (T2I) AI models, product designers have begun experimenting with them in their work. However, T2I models struggle to interpret abstract language and the current user experience of T2I tools can induce design fixation rather than a more iterative, exploratory process. To address these challenges, we developed Inkspire, a sketch-driven tool that supports designers in prototyping product design concepts with analogical inspirations and a complete sketch-to-design-to-sketch feedback loop. To inform the design of Inkspire, we conducted an exchange session with designers and distilled design goals for improving T2I interactions. In a within-subjects study comparing Inkspire to ControlNet, we found that Inkspire supported designers with more inspiration and exploration of design ideas, and improved aspects of the co-creative process by allowing designers to effectively grasp the current state of the AI to guide it towards novel design intentions.
David Chuan-En Lin, Hyeonsu B. Kang, Nikolas Martelaro, Aniket Kittur, Yan-Ying Chen, Matthew K. Hong
CHI3
2025 CompAct: Designing Interconnected Compliant Mechanisms with Targeted Actuation Transmissions
abstract
Compliant mechanisms enable the creation of compact and easy-to-fabricate devices for tangible interaction. This work explores interconnected compliant mechanisms consisting of multiple joints and rigid bodies to transmit and process displacements as signals that result from physical interactions. As these devices are difficult to design due to their vast and complex design space, we developed a graph-based design algorithm and computational tool to help users program and customize such computational functions and procedurally model physical designs. When combined with active materials with actuation and sensing capabilities, these devices can also render and detect haptic interaction. Our design examples demonstrate the tool's capability to respond to relevant HCI concepts, including building modular physical interface toolkits, encrypting tangible interactions, and customizing user augmentation for accessibility. We believe the tool will facilitate the generation of new interfaces with enriched affordance.
Humphrey Yang, I-Chao Shen, Nikolas Martelaro, Bo Zhu 0002, Haoran Xie 0002, Takeo Igarashi, Lining Yao
CHI3
2025 Punctuated and Prolonged: A Workers' Inquiry into Infrastructural Failures in Bus Transit
abstract
In North America, bus operators are essential but undervalued public servants — the ''human infrastructure'' of public transit. Transit workers face a range of largely invisible health and safety issues that have worsened in recent years. As more attention is directed toward new technologies being commercialized in the sector these operational challenges remain largely unaddressed. Our paper contributes to a turn ''back to labor'' and describes issues bus operators face on the job. Through a diary study of bus operators' working conditions we detail how punctuated moments of workplace violence, inhumane scheduling, and unsafe operational conditions become prolonged infrastructural failure. We outline how CSCW researchers and practitioners can contribute to the design of transit systems that enhance worker dignity and contribute to ongoing efforts to address urgent health and safety concerns.
Hunter Akridge, Alice Xiaodi Tang, Nikolas Martelaro, Sarah E. Fox
Proc. ACM Hum. Comput. Interact.3
2025 Understanding the Challenges of Maker Entrepreneurship
abstract
The maker movement embodies a resurgence in DIY creation, merging physical craftsmanship and arts with digital technology support. However, mere technological skills and creativity are insufficient for economically and psychologically sustainable practice. By illuminating and smoothing the path from "maker" to "maker entrepreneur," we can help broaden the viability of making as a livelihood. Our research centers on makers who design, produce, and sell physical goods. In this work, we explore the transition to entrepreneurship for these makers and how technology can facilitate this transition online and offline. We present results from interviews with 20 USA-based maker entrepreneurs (i.e., lamps, stickers), six creative service entrepreneurs (i.e., photographers, fabrication), and seven support personnel (i.e., art curator, incubator director). Our findings reveal that many maker entrepreneurs 1) are makers first and entrepreneurs second; 2) struggle with business logistics and learn business skills as they go; and 3) are motivated by non-monetary values. We discuss training and technology-based design implications and opportunities for addressing challenges in developing economically sustainable businesses around making.
Natalie Friedman, Alexandra Bremers, Adelaide Nyanyo, Ian Clark, Yasmine Kotturi, Laura A. Dabbish, Wendy Ju, Nikolas Martelaro
Proc. ACM Hum. Comput. Interact.8
2024 VideoMap: Supporting Video Exploration, Brainstorming, and Prototyping in the Latent Space
abstract
Video editing is a creative and complex endeavor and we believe that there is potential for reimagining a new video editing interface to better support the creative and exploratory nature of video editing. We take inspiration from latent space exploration tools that help users find patterns and connections within complex datasets. We present VideoMap, a proof-of-concept video editing interface that operates on video frames projected onto a latent space. We support intuitive navigation through map-inspired navigational elements and facilitate transitioning between different latent spaces through swappable lenses. We built three VideoMap components to support editors in three common video tasks. In a user study with both professionals and non-professionals, editors found that VideoMap helps reduce grunt work, offers a user-friendly experience, provides an inspirational way of editing, and effectively supports the exploratory nature of video editing. We further demonstrate the versatility of VideoMap by implementing three extended applications. For interactive examples, we invite you to visit our project page: https://chuanenlin.com/videomap.
David Chuan-En Lin, Fabian Caba Heilbron, Joon-Young Lee, Oliver Wang, Nikolas Martelaro
Creativity & Cognition5
2024 Videogenic: Identifying Highlight Moments in Videos with Professional Photographs as a Prior
abstract
This paper investigates the challenge of extracting highlight moments from videos. To perform this task, we need to understand what constitutes a highlight for arbitrary video domains while at the same time being able to scale across different domains. Our key insight is that photographs taken by photographers tend to capture the most remarkable or photogenic moments of an activity. Drawing on this insight, we present Videogenic, a technique capable of creating domain-specific highlight videos for a diverse range of domains. In a human evaluation study (N=50), we show that a high-quality photograph collection combined with CLIP-based retrieval (which uses a neural network with semantic knowledge of images) can serve as an excellent prior for finding video highlights. In a within-subjects expert study (N=12), we demonstrate the usefulness of Videogenic in helping video editors create highlight videos with lighter workload, shorter task completion time, and better usability.
David Chuan-En Lin, Fabian Caba Heilbron, Joon-Young Lee, Oliver Wang, Nikolas Martelaro
Creativity & Cognition5
2024 "The bus is nothing without us": Making Visible the Labor of Bus Operators amid the Ongoing Push Towards Transit Automation
abstract
This paper describes how the circumstances bus operators manage presents unique challenges to the feasibility of high-level automation in public transit. Avoiding an overly rationalized view of bus operators’ labor is critical to ensure the introduction of automation technologies does not compromise public wellbeing, the dignity of transit workers, or the integrity of critical public infrastructure. Our findings from a group interview study show that bus operators take on work — undervalued by those advancing automation technologies — to ensure the well-being of passengers and communities. Notably, bus operators are positioned to function as shock absorbers during social crises in their communities and in moments of technological breakdown as new systems come on board. These roles present a critical argument against the rapid push toward driverless automation in public transit. We conclude by identifying opportunities for participatory design and collaborative human-machine teaming for a more just future of transit.
Hunter Akridge, Bonnie Fan, Alice Xiaodi Tang, Chinar Mehta, Nikolas Martelaro, Sarah E. Fox
CHI5
2024 Co-design Accessible Public Robots: Insights from People with Mobility Disability, Robotic Practitioners and Their Collaborations
abstract
Sidewalk robots are increasingly common across the globe. Yet, their operation on public paths poses challenges for people with mobility disabilities (PwMD) who face barriers to accessibility, such as insufficient curb cuts. We interviewed 15 PwMD to understand how they perceive sidewalk robots. Findings indicated that PwMD feel they have to compete for space on the sidewalk when robots are introduced. We next interviewed eight robotics practitioners to learn about their attitudes towards accessibility. Practitioners described how issues often stem from robotic companies addressing accessibility only after problems arise. Both interview groups underscored the importance of integrating accessibility from the outset. Building on this finding, we held four co-design workshops with PwMD and practitioners in pairs. These convenings brought to bear accessibility needs around robots operating in public spaces and in the public interest. Our study aims to set the stage for a more inclusive future around public service robots.
Howard Ziyu Han, Franklin Mingzhe Li, Alesandra Baca-Vázquez, Daragh Byrne, Nikolas Martelaro, Sarah E. Fox
CHI5
2024 Jigsaw: Supporting Designers to Prototype Multimodal Applications by Chaining AI Foundation Models
abstract
Recent advancements in AI foundation models have made it possible for them to be utilized off-the-shelf for creative tasks, including ideating design concepts or generating visual prototypes. However, integrating these models into the creative process can be challenging as they often exist as standalone applications tailored to specific tasks. To address this challenge, we introduce Jigsaw, a prototype system that employs puzzle pieces as metaphors to represent foundation models. Jigsaw allows designers to combine different foundation model capabilities across various modalities by assembling compatible puzzle pieces. To inform the design of Jigsaw, we interviewed ten designers and distilled design goals. In a user study, we showed that Jigsaw enhanced designers’ understanding of available foundation model capabilities, provided guidance on combining capabilities across different modalities and tasks, and served as a canvas to support design exploration, prototyping, and documentation.
David Chuan-En Lin, Nikolas Martelaro
CHI2
2024 Power and Play: Investigating "License to Critique" in Teams' AI Ethics Discussions
abstract
Past work has sought to design AI ethics interventions--such as checklists or toolkits--to help practitioners design more ethical AI systems. However, other work demonstrates how these interventions may instead serve to limit critique to that addressed within the intervention, while rendering broader concerns illegitimate. In this paper, drawing on work examining how standards enact discursive closure and how power relations affect whether and how people raise critique, we recruit three corporate teams, and one activist team, each with prior context working with one another, to play a game designed to trigger broad discussion around AI ethics. We use this as a point of contrast to trigger reflection on their teams' past discussions, examining factors which may affect their ''license to critique'' in AI ethics discussions. We then report on how particular affordances of this game may influence discussion, and find that the hypothetical context created in the game is unlikely to be a viable mechanism for real world change. We discuss how power dynamics within a group and notions of ''scope'' affect whether people may be willing to raise critique in AI ethics discussions, and discuss our finding that games are unlikely to enable direct changes to products or practice, but may be more likely to allow members to find critically-aligned allies for future collective action.
David Gray Widder, Laura A. Dabbish, James D. Herbsleb, Nikolas Martelaro
Proc. ACM Hum. Comput. Interact.4
2023 The Robot in Our Path: Investigating the Perceptions of People with Motor Disabilities on Navigating Public Space Alongside Sidewalk Robots
abstract
Sidewalk robots are becoming increasingly common worldwide, yet their operation on public walkways presents challenges for pedestrians. This is especially true for people with motor disabilities (PWMD) who already manage obstacles such as inadequate ramps and public incivility. The addition of sidewalk robots could further intensify these difficulties, which poses an urgent need to examine how the design of sidewalk robots may influence the daily navigation experiences of PWMD. This poster illustrates findings from semi-structured interviews with ten PWMD, providing insights into their perspectives on the presence of sidewalk robots. The study uncovers potential conflicts in shared sidewalk use and the adaptive actions PWMD described needing to undertake in response. Interviewees raised concerns about whether the robots could accommodate the needs of PWMD, as compared to people walking on foot, and the repercussions of any shortcomings in this regard. Our research also examines tensions stemming from different robotic design choices, indicating the necessity for more accessible public robot designs. We further delve into PWMD’s interaction needs and modalities for routine operation and in the event of robot malfunction. As cities increasingly allow for the deployment of robots in public spaces, this work seeks to inform equitable design and deployment guidelines for sidewalk robots and calls for further research into the implications of the rise of public robots for the diverse populations that make up any given municipality.
Howard Ziyu Han, Franklin Mingzhe Li, Nikolas Martelaro, Daragh Byrne, Sarah E. Fox
ASSETS3
2023 Generative Image AI Using Design Sketches as input: Opportunities and Challenges
abstract
Generative image AI has sparked heated discussion among creative professionals. However, how it might be a tool for design practice is understudied, and it is not yet understood how designers may find image AI helpful across the stages of design practice. To address this, our preliminary study explores how designers use generative image AI accompanied by design sketches to inform early-stage 3D design. Further, we also examine the perceived limitations of text-to-image models. To do this, we recruited 11 Architecture graduate students with a median work experience of 2 years. Participants completed a design task using generative image AI packages and incorporated design sketches as inputs. The study findings provide insights into how image AI can or can not be a valuable resource for architectural design practitioners. Further, findings suggest possible directions for future image AI-assisted design tools and workflows.
Weijie Wang 0009, Paul Pangaro, Nikolas Martelaro, Daragh Byrne
Creativity & Cognition4
2023 Exploring Challenges and Opportunities to Support Designers in Learning to Co-create with AI-based Manufacturing Design Tools
abstract
AI-based design tools are proliferating in professional software to assist engineering and industrial designers in complex manufacturing and design tasks. These tools take on more agentic roles than traditional computer-aided design tools and are often portrayed as “co-creators.” Yet, working effectively with such systems requires different skills than working with complex CAD tools alone. To date, we know little about how engineering designers learn to work with AI-based design tools. In this study, we observed trained designers as they learned to work with two AI-based tools on a realistic design task. We find that designers face many challenges in learning to effectively co-create with current systems, including challenges in understanding and adjusting AI outputs and in communicating their design goals. Based on our findings, we highlight several design opportunities to better support designer-AI co-creation.
Frederic Gmeiner, Humphrey Yang, Lining Yao, Kenneth Holstein, Nikolas Martelaro
CHI5
2023 Soundify: Matching Sound Effects to Video
abstract
In the art of video editing, sound helps add character to an object and immerse the viewer within a space. Through formative interviews with professional editors (N=10), we found that the task of adding sounds to video can be challenging. This paper presents Soundify, a system that assists editors in matching sounds to video. Given a video, Soundify identifies matching sounds, synchronizes the sounds to the video, and dynamically adjusts panning and volume to create spatial audio. In a human evaluation study (N=889), we show that Soundify is capable of matching sounds to video out-of-the-box for a diverse range of audio categories. In a within-subjects expert study (N=12), we demonstrate the usefulness of Soundify in helping video editors match sounds to video with lighter workload, reduced task completion time, and improved usability.
David Chuan-En Lin, Anastasis Germanidis, Cristobal Valenzuela, Nikolas Martelaro
UIST5
2021 Learning Personal Style from Few Examples
abstract
A key task in design work is grasping the client’s implicit tastes. Designers often do this based on a set of examples from the client. However, recognizing a common pattern among many intertwining variables such as color, texture, and layout and synthesizing them into a composite preference can be challenging. In this paper, we leverage the pattern recognition capability of computational models to aid in this task. We offer a set of principles for computationally learning personal style. The principles are manifested in PseudoClient, a deep learning framework that learns a computational model for personal graphic design style from only a handful of examples. In several experiments, we found that PseudoClient achieves a 79.40% accuracy with only five positive and negative examples, outperforming several alternative methods. Finally, we discuss how PseudoClient can be utilized as a building block to support the development of future design applications.
David Chuan-En Lin, Nikolas Martelaro
Conference on Designing Interactive Systems2
2021 Leveraging the Twitch Platform and Gamification to Generate Home Audio Datasets
abstract
Training AI systems requires large datasets. While there are a range of existing methods for collecting such data, such as paid work on crowdsourcing platforms, the strengths and weaknesses of each method leads us to believe that new, complementary methods are needed. The Polyphonic project contributes a novel method for collecting real-world data by piggybacking on game streaming communities such as Twitch, which capture over a trillion minutes of viewer attention a year. By embedding activities within the sociotechnical context of the stream, we can leverage some of this attention for data collection and processing. In this paper, we describe the design and implementation of a proof-of-concept system for collecting home audio data. We conducted a field study in four live streams and found that our proof-of-concept effectively supports data capture. We also contribute further design insights about stream-based data collection systems.
Nikolas Martelaro, Tarannum Lakdawala, Jessica Hammer
Conference on Designing Interactive Systems1
2020 Using Remote Controlled Speech Agents to Explore Music Experience in Context
abstract
It can be difficult for user researchers to explore how people might interact with interactive systems in everyday contexts; time and space limitations make it hard to be present everywhere that technology is used. Digital music services are one domain where designing for context is important given the myriad places people listen to music. One novel method to help design researchers embed themselves in everyday contexts is through remote-controlled speech agents. This paper describes a practitioner-centered case study of music service interaction researchers using a remote-controlled speech agent, called DJ Bot, to explore people's music interaction in the car and the home. DJ Bot allowed the team to conduct remote user research and contextual inquiry and to quickly explore new interactions. However, challenges using a remote speech-agent arose when adapting DJ Bot from the constrained environment of the car to the unconstrained home environment.
Nikolas Martelaro, Sarah Mennicken, Jennifer Thom-Santelli, Henriette Cramer, Wendy Ju
Conference on Designing Interactive Systems1
2019 An Exploration of Speech-Based Productivity Support in the Car
abstract
In-car intelligent assistants offer the opportunity to help drivers productively use previously unclaimed time during their commute. However, engaging in secondary tasks can reduce attention on driving and thus may affect road safety. Any interface used while driving, even if speech-based, cannot consider non-driving tasks in isolation of driving---alerts for safer driving and timing of the non-driving tasks are crucial to maintaining safety. In this work, we explore experiences with a speech-based assistant that attempts to help drivers safely complete complex productivity tasks. Via a controlled simulator study, we look at how level of support and road context alerts from the assistant influence a driver's ability to drive safely while writing a document or creating slides via speech. Our results suggest ways to support speech-based productivity interactions and how speech-based road context alerts may influence driver behavior.
Nikolas Martelaro, Jaime Teevan, Shamsi T. Iqbal
CHI1
2019 Is Now A Good Time?: An Empirical Study of Vehicle-Driver Communication Timing
abstract
Advances in automotive sensing systems and speech interfaces provide new opportunities for smarter driving assistants or infotainment systems. For both safety and consumer satisfaction reasons, any new system which interacts with drivers must do so at appropriate times. We asked 63 drivers, ''Is now a good time?'' to receive non-driving information during a 50-minute drive. We analyzed 2,734 responses and synchronized automotive and video data, and show that while the chances of choosing a good time can be determined with better success using easily accessible automotive data, certain nuances in the problem require a richer understanding of the driver and environment states in order to achieve higher performance. We illustrate several of these nuances with quantitative and qualitative analyses to contribute to the understanding of how to design a system that might simultaneously minimize the risk of interacting at a bad time while maximizing the window of allowable interruption.
Rob Semmens, Nikolas Martelaro, Pushyami Kaveti, Simon Stent, Wendy Ju
CHI2
2017 Toward Measurement of Situation Awareness in Autonomous Vehicles
abstract
Until vehicles are fully autonomous, safety, legal and ethical obligations require that drivers remain aware of the driving situation. Key decisions about whether a driver can take over when the vehicle is confused, or its capabilities are degraded, depend on understanding whether he or she is responsive and aware of external conditions. The leading techniques for measuring situation awareness in simulated environments are ill-suited to autonomous driving scenarios, and particularly to on-road testing. We have developed a technique, named Daze, to measure situation awareness through real-time, in-situ event alerts. The technique is ecologically valid: it resembles applications people use in actual driving. It is also flexible: it can be used in both simulator and on-road research settings. We performed simulator-based and on-road test deployments to (a) check that Daze could characterize drivers' awareness of their immediate environment and (b) understand practical aspects of the technique's use. Our contributions include the Daze technique, examples of collected data, and ways to analyze such data.
David Sirkin, Nikolas Martelaro, Mishel Johns, Wendy Ju
CHI2
2017 WoZ Way: Enabling Real-time Remote Interaction Prototyping & Observation in On-road Vehicles
abstract
Interaction designers often have difficulty understanding people's real-world experiences with ubiquitous systems. The automobile is a great example of these challenges, where on-road testing is time-consuming and provides little ability for rapid prototyping of interface behavior. We introduce WoZ Way, a system to connect designers to remote drivers. We use live video, audio, car data, and Wizard of Oz speech and interfaces to enable remote observation and interaction prototyping on the road. Our implementation integrates environmental, system level, and social information to make the invisible visible. We tested across three example deployments highlighting usage in interaction prototyping and observational studies. Our findings illustrate how designers explored the situated experiences of people on the road, and how they experimented with different improvisational Wizard of Oz interactions. WoZ Way is both a design research and a design prototyping tool, which can support the work of interaction designers through naturalistic observations, contextual inquiry, and responsive interaction prototyping.
Nikolas Martelaro, Wendy Ju
CSCW1
2017 Making Noise Intentional: A Study of Servo Sound Perception
abstract
How do sounds shape interaction with robots? The present study explores aural impressions associated with servo motors commonly used to prototype robotic motion. This exploratory analysis constructs a framework to objectively and subjectively characterize sound using acoustic analyses and novice evaluators on Amazon Mechanical Turk. Participants evaluated unfamiliar sounds through pairwise comparison, resulting in subjective ratings of servo motor sounds. In this study, subjective measures of sound correlated well internally, but correlated weakly with objective measures. Moreover, qualitative commentary offered by participants suggests both anthropomorphic associations with sounds as well as negative impressions of the sounds overall. We conclude with a roadmap for exploration into the field of consequential sonic interaction design.
Dylan Moore 0001, Hamish Tennent, Nikolas Martelaro, Wendy Ju
HRI3
2016 Tell Me More: Designing HRI to Encourage More Trust, Disclosure, and Companionship
abstract
Previous HRI research has established that trust, disclosure, and a sense of companionship lead to positive outcomes. In this study, we extend existing work by exploring behavioral approaches to increasing these three aspects of HRI. We increased the expressivity and vulnerability of a robot and measured the effects on trust, disclosure, and companionship during human-robot interaction. We engaged (N = 61) high school aged students in a 2 (vulnerability of robot: high vs. low) × 2 (expressivity of robot: high vs. low) between-subjects study where participants engaged in a short electronics learning activity with a robotic tutor. Our results show that students had more trust and feelings of companionship with a vulnerable robot, and reported disclosing more with an expressive robot. Additionally, we found that trust mediated the relationship between vulnerability and companionship. These findings suggest that vulnerability and expressivity may improve peoples' relationships with robots, but that they each have different effects.
Nikolas Martelaro, Victoria Chibuogu Nneji, Wendy Ju, Pamela J. Hinds
HRI1
2016 Tell Me More: Designing HRI to encourage more trust, disclosure, and companionship
abstract
Previous HRI research has established that trust, disclosure, and a sense of companionship lead to positive outcomes. We explored improving these aspects of HRI through robot behavior. Specifically, we increased the expressivity and vulnerability of a robot and measured the effects on trust, disclosure, and companionship during human-robot interaction. We engaged (N = 61) high school aged students in a 2 (vulnerability of robot: high vs. low) × 2 (expressivity of robot: high vs. low) between-subjects study where participants engaged in a short electronics learning activity with a robotic tutor. Our results show that students had more trust and feelings of companionship with a vulnerable robot, and reported disclosing more with an expressive robot. This video highlights some of the qualitative interactions students had with the robot as well as a discussion of their experiences around trust, disclosure, and companionship with the robot. The video clips of student interactions show how vulnerability and expressivity can be used to engender trust, disclosure, and companionship between a person and robot.
Nikolas Martelaro, Victoria Chibuogu Nneji, Wendy Ju, Pamela J. Hinds
HRI1
2016 Design Skills for HRI
abstract
This tutorial is a hands-on introduction to human-centered design topics and practices for human-robot interaction. It is intended for researchers with a variety of backgrounds, particularly those with little or no prior experience in design. In the morning, participants will learn about user needs and needfinding, as ways to understand the stakeholders in research outcomes, guide the selection of participants, and as possible measures of success. We then focus on design sketching, including ways to represent objects, people and their interactions through storyboards. Design sketching is not intended to be art, rather a way to develop and build upon ideas with oneself, and quickly communicate with colleagues. In the afternoon, participants will use the tools and materials, and learn techniques for lightweight physical prototyping and improvisation. Participants will build a small paper robot (not actuated) of their own design, to practice puppeteering, explore bodily movement and prototype interactions.
David Sirkin, Nikolas Martelaro, Hamish Tennent, Mishel Johns, Brian K. Mok, Wendy Ju, Guy Hoffman, Heather Knight, Bilge Mutlu, Leila Takayama
HRI2
2016 The Interaction Engine: Tools for Prototyping Connected Devices
abstract
In this workshop, we will familiarize participants with the Interaction Engine, a system for prototyping connected, interactive devices using low cost, single-board Linux computers and Arduino microcontrollers. Our main objective is to introduce participants to the basic architecture of connected devices and provide hands-on experience creating networked, physical hardware. The Interaction Engine is a generic framework, not a specialized toolkit. We employ widely available, community-supported tools that can enable web-connected hardware capable of merging tangible interfaces with audio/visual web interfaces. We view low-cost single-board computers as an enabling technology, representing the next step for tangible, embedded, and embodied designs enabling deep interaction between physical and digital worlds. This workshop will be a starting point for participants to begin exploring connected device development and will provide a basic set of tools and skills that participants can use in their own applications.
Nikolas Martelaro, Michael Shiloh, Wendy Ju
TEI1
2016 Designing the Behavior of Interactive Objects
abstract
To design proactive and autonomous interactive objects, designers deal with the design of the object's behavior. In this paper, we propose a design method, called Personality, to help designers develop interactive objects' behaviors with a focus on aesthetics of interaction; the method focuses on tangible and bodily interaction, and it includes four main steps. The "unguided improvisation" step consists of an initial interplay with the interactive object in order to size up the interaction; a brainstorming step, in which we use stereotypes of personalities to create metaphors, to support the discussion around, and the description of, possible behaviors; the "guided improvisation" step iterates over several improvisation sessions to act out interaction scenarios and behaviors; and the behavior synthesis step, in which we provide a final description of the object's behavior. To illustrate Personality we will describe the sofa-bot case study. We will report a lab study, in which we observed people reaction to the different behaviors of the sofa.
Marco Spadafora, Victor Chahuneau, Nikolas Martelaro, David Sirkin, Wendy Ju
TEI3
2015 The RRADS platform: a real road autonomous driving simulator
abstract
This platform paper introduces a methodology for simulating an autonomous vehicle on open public roads. The paper outlines the technology and protocol needed for running these simulations, and describes an instance where the Real Road Autonomous Driving Simulator (RRADS) was used to evaluate 3 prototypes in a between-participant study design. 35 participants were interviewed at length before and after entering the RRADS. Although our study did not use overt deception---the consent form clearly states that a licensed driver is operating the vehicle---the protocol was designed to support suspension of disbelief. Several participants who did not read the consent form clearly strongly believed that they were interacting with a fully autonomous vehicle.
Sonia Baltodano, Srinath Sibi, Nikolas Martelaro, Nikhil Gowda, Wendy Ju
AutomotiveUI3
2015 Using Robots to Moderate Team Conflict: The Case of Repairing Violations
abstract
We explore whether robots can positively influence conflict dynamics by repairing interpersonal violations that occur during a team-based problem-solving task. In a 2 (negative trigger: task- directed vs. personal attack) x 2 (repair: yes vs. no) between- subjects experiment (N = 57 teams, 114 participants), we studied the effect of a robot intervention on affect, perceptions of conflict, perceptions of team members' contributions, and team performance during a problem-solving task. Specifically, the robot either intervened by repairing a task-directed or personal attack by a confederate or did not intervene. Contrary to our expectations, we found that the robot's repair interventions increased the groups' awareness of conflict after the occurrence of a personal attack thereby acting against the groups' tendency to suppress the conflict. These findings suggest that repair heightened awareness of a normative violation. Overall, our results provide support for the idea that robots can aid team functioning by regulating core team processes such as conflict.
Malte F. Jung, Nikolas Martelaro, Pamela J. Hinds
HRI2
2014 Participatory materials: having a reflective conversation with an artifact in the making
abstract
Designing and building mechatronic systems has gradually ceased to be the domain of only highly trained professionals and has become broadly accessible. Drawing from a notion of designing as a conversation with the materials of the situation we built an artifact that could actively engage in its own making by embedding a Wizard of Oz operated animated agent into an Arduino prototyping platform. In a 2x2 between-participants Wizard of Oz laboratory experiment with (N=68) high-school students we specifically examined how this prototyping agent's expression of interest affected perceptions of the agent and learning outcomes dependent on the embodiment of the agent as embedded in the prototyping material itself or as an external entity. We found evidence that embedding an agent into the prototyping material can positively influence learning processes and outcomes while not harming perceptions of the agent.
Malte F. Jung, Nikolas Martelaro, Halsey Hoster, Clifford Nass
Conference on Designing Interactive Systems2