While generative AI has gained immense popularity, its application in specialized fields, particularly in music production, remains challenging. Little research has been done on how music producers can most intuitively communicate with generative AI to co-produce music. In this paper, we explored how to employ gestural and vocal interactions for an intuitive human-AI music co-production process. Through task analysis of computer-assisted music production, online surveys probing the key challenges in music production, and an in-lab gesture elicitation study with 14 music producers, we uncovered how music producers employ 9 types of gestures (e.g., finger snapping, body shaking, etc.) accompanied by 2 types of vocal expressions (i.e., beatboxing and humming) to interact with a music-generating AI system, communicating 4 music features during music production. This research advances understanding of embodied interactions between music producers and generative AI, while demonstrating how gesture elicitation data from open-ended user requests can be analyzed.

“Space Agents” refers to intelligent and interactive agents that are both environmental (i.e. capable of reconfiguring virtual or physical spaces/environments) and human-like in (at least some of) their behavior. This paper builds upon the previous work from Sharmayne et al. of the sphere-based Virtual Reality (VR) space agent (a space agent in virtual space), and proposes the design of a mist-augmented VR space agent – an intelligent VR space that interacts with designers through weather changes (i.e., movement and density of mist) in VR. The design of this mist-augmented VR space agent is generated from co-design workshops aiming to address the limitations of the sphere-based VR space agent in the aspect of promoting human- “space agent” design partnership. Next, the mist-augmented and sphere-based VR space agents are compared, evaluating their effectiveness in 1) evoking users’ social perception of a VR space agent as a design partner, 2) supporting users’ design process in VR, and 3) promoting users’ creativity. The results suggest that the mist-augmented space agent is better in all three aspects. This paper explores the novel topic of designing weather conditions in VR as an interactive medium to enable human- “space agent” social interactions, and empirically shows the potential of human- “VR space agent” design partnerships for promoting designers’ creativity.

“Socially Interactive Virtual Environment” (SIVE) refers to a reconfigurable virtual environment that is perceived and interacted with as a socially intelligent agent. SIVE is a subset of “Space Agents”, which refers to intelligent or interactive agents that are both environmental (i.e. capable of reconfiguring virtual or physical spaces/environments) and human-like in (at least some of) their behavior. In this paper, we first proposed the concept of SIVE, and then explored the intricate human-SIVE interactions in the context of design partnership through empirical studies. More specifically, through the empirical studies, we aim to gain a preliminary understanding of how users perceive SIVE's social and environmental communications when SIVE is trying to play the role of a design partner, and how such user perceptions may shape the human-SIVE design partnership. Harnessing Unreal Engine, we designed and developed a virtual environment that used various visual and auditory modalities to interact with users and employed the Wizard-of-Oz (WoZ) technique to conduct in-lab experiments with 10 participants. The participants' perspectives and interactions were video recorded, and then carefully reviewed and analyzed by multiple researchers. A semi-structured interview was also conducted with each participant after the WoZ experiment, with the transcripts analyzed through the grounded theory coding technique. The findings of this study contribute to a preliminary understanding of the potential benefits and challenges associated with integrating SIVE into collaborative design practices and have the potential to shape the future of human- “intelligent agent” design partnerships and co-creations, offering guidance for designers, developers, and researchers seeking to leverage the social expressiveness of a virtual space agent as a design partner.

Could robotic furnishings become reading companions for children? This paper investigates how a socially interactive robotic bookshelf may influence children’s enthusiasm, persistence, and enjoyment in reading when they are absorbed in other, usually more addictive activities such as playing iPad games or toys. We designed and developed a low-fidelity, elephant-like robotic bookshelf, and used WoZ technique to conduct an experiment in a public library in Singapore with 7 child-parent pairs to assess the effectiveness of this socially interactive, mobile robotic bookshelf in attracting children to read physical books while iPad games and toys were present. The results indicate that the robotic bookshelf was successful in attracting 5 out of 7 children to read books, with variations observed in the timing of switching to book reading from playing iPad games or toys. Currently, most robot reading companions are tabletop robots passively waiting for children to come to the robots and read with them. This study, however, highlights a unique approach in which a mobile, robotic bookshelf reading companion proactively engages children in reading by bringing physical books to them. In a modern world where screen and other addictions become increasingly real for children, such mobile robotic bookshelves have great potential to benefit children’s literacy and overall development.

Architecture has long been conceptualized as “a machine for living in” and more recently as “a robot for living in.” Human-Robot Interaction (HRI) has developed robots as social agents—our friends, companions, and partners. Could robotic environments be perceived and interacted with as socially intelligent agents? If so, how should we design a Socially Interactive, Robotic Environment (SIRE)? To address the first question, we offer the empirical evidence and theoretical support of SIREs. We then address the second question by discussing the “Spatial Design” and “Interaction Design” of SIREs through an explorative, pattern-based approach. For “Spatial Design,” we present a co-design study for a partner-like office, generating new spatial patterns that form pattern languages to convey sociality to individual users. For “Interaction Design,” we employed four “Design Patterns for Sociality in HRI.” Our results show that “Spatial Patterns” and “HRI Patterns” can be integrated as one pattern language for sociality and that such a pattern language can vary from person to person. Through the explorative works of this paper, we wish to introduce SIRE to IE communities and cultivate the conversation about the design and application of SIREs in everyday life.

Novel, "space-making" robots have potential to redefine physical space and the human activities occurring in it. Categorically distinct from many robots and far removed from humanoids, space-making robots are not objects in space, not anthropomorphic, not animal-like, not mobile, but instead, integral with the physical environment, embedded in or forming walls, ceilings, floors, partitions, vehicle interiors, and building envelopes. Given their distinctiveness, space-making robots offer a novel human-machine interaction. This paper investigates whether users perceive space-making robots as agents- artificial social actors characterized by the capacity for intelligence, recognition, and intention. Results of an in-lab experiment with 11 participants and an online, between-group experiment with 120 participants show that people attribute agency metrics of intelligence, intention, recognition, cooperation, collaboration, friendliness, and welcome to our reconfigurable robotic surface embedded in a wall partition. While space-making robots may become numerous in the built environment, our results are significant, moreover, for their broader implications for conceptualizing and designing human-machine interactions.

Developments of robot arms are countless, but there has been little focus on robot surfaces for the reshaping of a habitable space—especially compliant surfaces. In this paper we introduce a novel, tendon-driven, robot surface comprised of aggregated, overlapping panels organized in a herringbone pattern. The individual 3D-printed panels and their behavior as an aggregation are inspired by the form and behavior of a pinecone. This paper presents our concept, design, and realization of this robot, and compares our prototype to simulations of four physical configurations that are formally distinct and suggestive of how the surface might be applied to habitable, physical space in response to human needs and wants. For the four configurations studied, we found a validating match between prototype and simulations. The paper concludes with a consideration of potential applications for robot surfaces like this one.

This paper presents the rationale and current progress of my Ph.D. dissertation: "design interactions between robot surfaces and human designers." This specific topic serves as a case study trying to explore the question of how to design an interactive and partially intelligent space. We proposed the concept of "space agent" defined as "interactive and intelligent environments perceived by users as human agents" based on communication theories. Built upon this concept, we proposed a design framework for interactive environments. Then we further explored literatures about what space agent could contribute to human users specifically for the case of interior designers' work space. Research questions and research designs are introduced in this paper, followed by the discussions of experiments design.

Communication theory suggests that people tend to interact with interactive artifacts as if these were human. For decades, this understanding has been applied to designing singular, embedded artifacts at a small physical scale. In this paper, we extend the same theory and practice to the dimension of space-to designing interactive, physical environments and their components. A conceptual ground for this is found in a "pattern language" developed by Alexander et al. for designing static physical environments. Upon this ground, we construct a systematic framework for designing "collaborative environments" shaped, as well, by our own concepts, Direct Mapping, Conveyed Mapping, and Space Agency, to strive for more human-human-like interactions between human beings and their physical surroundings. Our lab-based study generates a hypothetical design as qualitative validation of the framework, which has significance for designing tangible, embedded, and embodied interaction as it extends, inevitably, to the dimension of space, entertaining, serving, and augmenting us.