Interaction while performing physical tasks is inherently challenging, as both hands are fully engaged. In Minimally-Invasive Surgery (MIS), for instance, navigating images requires either delegation, causing frustration and delays, or hand de-sterilization, increasing risk. We introduce Interaction Through Instruments, an interaction paradigm in which task instruments become interaction devices. To design this technique in MIS, we first conduct a survey (N=23) identifying intraoperative needs, interaction strategies and workarounds, and persistent challenges. Then, through five participatory design workshops (N=10), we identify challenges in blending a user interface into views of a physical space, informing the design of InteractOR, a system that combines surgical instrument segmentation with pinch-gesture recognition to enable interaction within the surgical view. Finally, in a Comparative Structured Observation study (N=12) we compare two visualization strategies (side-by-side and overlay) against delegation, showing that interaction through instruments can reduce focus shifts, increase efficiency, and foster autonomy.

As AI systems increasingly display human-like behavior, people often attribute consciousness to these machines, raising questions about anthropomorphism, ethics, and design. We report findings from an online survey (N=553), including academics from the formal sciences, natural sciences, humanities, and a heterogeneous group of other participants. Respondents evaluated perceptions of consciousness in large language models and future AI, alongside related ethical and policy considerations. The results show that, across groups, around half of the participants attributed some degree of consciousness. Individual traits such as gender, academic position, and beliefs regarding consciousness and intelligence of systems strongly shape perceptions, outweighing the effects of technical knowledge or system transparency. Beyond shaping academic discussions, these perspectives inform how AI is designed, governed, and integrated into everyday interactions.

Current hospital medication management systems generate fragmented clinical workflows, forcing clinicians to improvise repairs by creating external artifacts. We argue that instead of formalizing their workarounds into new electronic health record (EHR) features, future systems should explicitly support improvisation capabilities that help clinicians better handle unpredictable breakdowns. In a field study (144 hours) with 11 clinicians, we observed how resource constraints require local optimizations — “expert hacks” by clinicians — at the expense of global consistency. Subsequent group interviews with 20 clinicians highlighted three distinct issues: one clinician’s prescribing shortcut often becomes another clinician’s roadblock; medication notes fail to distinguish between norms and deviations; and clinicians often reify personal strategies to translate medication data across systems. We argue that simply adding new electronic features will not suffice. Instead, we propose a novel design approach that focuses on helping clinicians create personal tools that let them successfully manage information breakdowns in their particular context.

Psychological ownership over virtual and physical spaces in augmented reality can lead to tensions between collaborators, yet, there is still a significant challenge in understanding how psychological ownership manifests in shared AR and what that might mean for the inclusion of collaborative interaction mechanisms. Through an experimental instruction task with a teleAR system, we interviewed 16 participant pairs on their perceptions of ownership of virtual and physical spaces and how they thought their perceptions impacted their interaction within those spaces. Our findings indicate (1) how AR introduces new ideas around behavioral norms in spaces that are layered and (2) that the nature of the task itself, in this case one of instruction where collaborators have different levels of knowledge and the local worker is reliant on the remote expert, significantly affects the perceptions of ownership and therefore behavior norms.

This study investigates how annotation control influences group processes in teleAR and examines the mediating role of psychological ownership. Through an experiment comparing three ego-centrically defined annotation control forms - We control (both annotate), I control (one annotates), and They control (partner annotates) - we found that ego-centric perceptions of annotation control significantly shaped collaborative dynamics. We control consistently showed advantages across all measured clusters of group processes, showing significant improvements over both I control and They control. Additionally, our analysis revealed that Collective Psychological Ownership (CPO) served as a crucial mediator between annotation control and group processes, particularly in comparisons between We control and They control. Further examination of the five psychological ownership dimensions revealed limited but notable mediation effects, with the Possession and Control dimensions showing positive indirect effects on Human development and Shared practices when comparing I control with They control. These findings show how annotation control forms influence distributed collaboration and highlight the importance of considering psychological ownership in improving group processes.

Surgery is primarily taught through mentoring, where an expert mentor supervises a mentee performing surgery, taking over when necessary. Telementoring systems aim to provide mentees with access to remote mentors, but the physical distance between mentors and mentees poses unique challenges to surgical training. We investigate the underlying needs leading to takeovers in onsite mentoring and assess mentors’ ability to fulfill address these needs remotely using existing telestration tools, namely pointers and drawings on shared views. Through interviews and workshops with expert surgeons, we find that (1) mentors take over to convey gestures related to instrument placement, tissue displacement, force, and movement, (2) mentors gather information about location of tissue, equipment, and instruments, as well as gesture constraints, and (3) surgeons judge telestration insufficient for these needs. Based on this gap between onsite mentoring practices and telementoring tools, we discuss novel tools to address these needs and their evaluation.

All along their careers, surgeons learn through mentoring, operating on patients side-by-side with expert surgeons. However, access to experts is more and more challenging as they are often located far from learners. Telementoring can increase the reach of mentors by supporting mentoring at a distance, but despite their potential, current systems remain rarely institutionalized. We investigate current surgical mentoring practices, to inform future design of telementoring systems in providing support for learning. In a field study, we observe 11 surgeries and conduct follow-up self-confrontation interviews with mentor-mentee pairs. Through Thematic Analysis, we find that the domain of surgery results in mentee's needs for reassurance regarding safety, being spared from workload and risk, both essential to support the acquisition of skills. Second, that mentors understand the situation primarily through the surgical site view but also through inspecting the mentee's posture and the patient's body. Thirdly, that mentors take physical distance as their presence limits mentees' autonomy development. We discuss the limits of current surgical telementoring systems and rethink their role as companions of onsite mentoring.

Although Augmented Reality (AR) has been touted as the future of surgery, its contribution to distributed collaboration such as in surgical teleconsulting has not been articulated. We propose AR-Head Mounted Displays (AR-HMD) to tackle two previously-identified challenges: operating surgeons needing to view and interact with imaging systems that reside away from the operative field, and, their lack of gesturing tools to point and annotate on the shared images and physical environment. We report on a controlled lab experiment where 12 expert gynecology surgeons perform a tumor localisation task guided by a remote radiologist (confederate) via an AR-HMD. We find that bringing the shared images to the place of work reduces the need for clarifications and provides opportunistic access to information when required, and, that pointing and annotating provides opportunities to further support verbal instruction in deictic communication. Our results inform the design of intraoperative AR-HMD systems for surgical telecollaboration.

Videos are a powerful media to learn activities through guided physical training such as surgery, especially when they are produced following human learning models and not as "how-to" videos. However, their success greatly depends on how they are integrated into the extensive curricula of domains where learning occurs through guided practice. In this work, we investigate the impact of integrating video as a learning tool into the learning curricula of surgery. We created a pedagogical video on surgical hysterectomy through a model based on the Conceptual Fields theory (Vergnaud) and performed two rounds of interviews with seven medical residents, who watched the video freely during their residency in gynecology-obstetrics as they trained with experts. We find that videos can complement guided physical training, as they can provide the rationale behind expert action, something that is difficult to explicit during guided training. Still, their linear and static nature limits their integration as true adjuncts. We discuss our vision of moving towards interactive videos created with an ontological approach, developed in a workshop with four expert surgeons, which involves the ability to navigate through levels of information and layers of representations, so that experts can represent information to learners according to pedagogical models that complement their complex and extensive learning curricula.

The introduction of Augmented Reality (AR) Head Mounted Displays (HMDs) in collaboration between remote and local workers, introduces new challenges given that camera views are now mobile. We introduce HoloMentor, an AR HMD-based collaborative system designed for remote instruction over live mobile views during physical tasks. Through Actionport, we provide a world-stabilized area where remote helpers can dynamically place a pointer and annotations on the physical environment. Through Actionpad, we provide an indirect input mechanism with an absolute position to the Actionport. We show how these innovations worked for participants engaged in a remote instructional task and how they supported effective and efficient communication. Finally, we provide the next steps for addressing AR on mobile views for remote instruction.

Capturing content of fabrication activities is the first step for producing knowledge resources and an integral part of maker culture. As a secondary task, it conflicts though with the fabrication activity, and thus it is often forgotten and knowledge resources end up incomplete. In this article, we investigate different dimensions of content capture for knowledge resources in fabrication workshops. Based on past work in this area, we first propose a framework through which we identify two research directions to investigate. From these, we derive three dimensions to explore in more depth: The number of capturing devices, their feature variety and the degree of automation of each feature. We then explore the design space resulting from these three dimensions with the help of a design concept and an online survey study (N=66). Results show (1) a variety of needs and preferences justifying feature variety and multiplicity, (2) challenges in defining the right degree of manual and automatic control, and (3) the socio-technical impact of cameras in a shared space regarding privacy and ethics. We conclude with discussions on the benefits and vulnerabilities of equipping fabrication workshop with distributed camera-based capturing devices and offer opportunities for design.

Telerehabilitation systems for stroke survivors have been predominantly designed to measure and quantify movement in order to guide and encourage rehabilitation regular exercises at home. We set out to study what aspect of the movement data was essential, to better inform sensor design. We investigated face-to-face stroke rehabilitation sessions through a series of interviews and observations involving 16 stroke rehabilitation specialists including physiatrists, physical therapists, and occupational therapists. We found that specialists are not solely interested in movement data, and that experiential information about stroke survivors’ lived experience plays an essential role in specialists interpreting movement data and creating a rehabilitation plan. We argue for a reconceptualization in stroke telerehabilitation that is more inclusive of non-movement data, and present design implications to better account for experiential information in telerehabilitation systems.

While surgical videos are valuable support material for activities around surgery, their summarization demands great amounts of time from surgeons, resulting in the production of very few videos. We study the practices involving surgical video to motivate and inform the future design of tools for their summarization. Through interviews and observations in a field study, we find that (1) video summaries provide an important support for surgery, being used for self-improvement, education, discussing cases, scientific research, patient communication and as legal resources; (2) video summarization follows a process hindered by the loss of knowledge that originates during recording; and, (3) surgeons develop ad-hoc coordination strategies which involve using the video itself for articulation work, making it both the field of work and coordination artifact. We discuss ways in which tools can facilitate capturing knowledge during live action using these strategies.

Bedside robotic endoscopes render surgeons autonomous from assistants, potentially improving surgical outcome and decreasing costs. Why then have they not been widely adopted? We take a step back and first characterize classic (non-robotic) endoscope use through observations, literature and a domain expert interview. We review the literature on bedside robotic endoscopes and find that existing controls, individually, do not have the power to support both intended and appropriated endoscope uses. We thus explore combining controls to support this diversity of uses. Through an iterative cycle, we design and implement a multimodal and mixed-initiative technique that combines two user controls and one system control. Our evaluations confirm that individual controls do not satisfy the diversity of endoscope uses, and also that our technique indeed does so. Our work highlights the relevance of HCI research in the medical domain through robotic systems.

Robotic-assisted Minimally Invasive Surgery (MIS) is adopted more and more as it overcomes the shortcomings of classic MIS for surgeons while keeping the benefits of small incisions for patients. However, introducing new technology oftentimes affects the work of skilled practitioners. Our goals are to investigate the impacts of telemanipulated surgical robots on the work practices of surgical teams and to understand their cause. We conducted a field study observing 21 surgeries, conducting 12 interviews and performing 3 data validation sessions with surgeons. Using Thematic Analysis, we find that physically separating surgeons from their teams makes them more autonomous, shifts their use of perceptual senses, and turns the surgeon's assistant into the robot's assistant. We open design opportunities for the HCI field by questioning the telemanipulated approach and discussing alternatives that keep surgeons on the surgical field.

Surgeons performing laparoscopic surgery experience distortion when perceiving the stiffness of a patient's tissues. This is due to the lever effect induced by the introduction of instruments in their patient's body through a fulcrum. To address this problem, we propose to use the comanipulation paradigm. A robotic device is connected to the handle of the instrument while simultaneously being held by the surgeon. This device applies a force on the handle that reflects the force measured at the tool tip, with a gain that depends on the lever ratio. The implementation of this method is presented on an experimental setup and a preliminary assessment experiment is presented.

Remote collaboration across wall-sized displays creates a key challenge: how to support audio-video communication among users as they move in front of the display. We present CamRay, a platform that uses camera arrays embedded in wall-sized displays to capture video of users and present it on remote displays according to the users' positions. We investigate two settings: in Follow-Remote, the position of the video window follows the position of the remote user; in Follow-Local, the video window always appears in front of the local user. We report the results of a controlled experiment showing that with Follow-Remote, participants are faster, use more deictic instructions, interpret them more accurately, and use fewer words. However, some participants preferred the virtual face-to-face created by Follow-Local when checking for their partners' understanding. We conclude with design recommendations to support remote collaboration across wall-sized displays.

This paper presents a controlled experiment assessing the accuracy when interpreting remote users showing a shared object on a large wall-sized display, either by looking at it or by looking and pointing at it. We analyze both distance and angle errors and how they are sensitive to the relative position be- tween the remote viewer and the video feed. We show that the remote user can accurately determine the target, that eye gaze alone is more accurate than combined with the hand, and that the relative position between the viewer and the video feed has little effect on accuracy. These findings can inform the design of future telepresence systems for wall-sized displays.