Multi-display environments (MDEs) are applicable to both everyday and specialized tasks like cooking, appliance repair, surgery, and more. In these settings, displays are often affixed in a manner that prevent reorientation, forcing users to split their attention between multiple visual information sources. Mixed reality (MR) has the potential to transform these spaces by presenting information through virtual interfaces that are not limited by physical constraints. While MR has been explored for single-task work, its role in multi-task, information-dense environments remains relatively unexplored. Our work bridges this gap by investigating the impact of different display modalities (large screens, tablets, and MR) on performance and perception in these environments. Our study's findings demonstrate the capability for MR to integrate into these spaces, extending traditional display technology with no impact to performance, cognitive load, or situational awareness. The study also further illustrates the nuanced relationship between performance and preference in tools used to guide task work. We provide insights toward the eventual authentic integration of MR in MDEs.

The rapid development and deployment of “smart space” technology in commercial and educational buildings has brought efficiencies and comfort to users. Yet, the “smart” is largely limited to knowledge gained from occupancy and activity data. These limited data explain what is happening in a space but nothing about how effectively the space is being used. In this work, we explore how to understand effective space use and how it may impact future smart spaces and their design. As part of this investigation, we design generalizable heuristics derived from observable human behaviors and space usage to detect group configurations in public spaces. The resultant system models similar qualities of a space that end-users will judge and measure themselves before use. The effectiveness of our system is validated against a crowdsourced study. Our system serves as a foundation for applications building upon improving space use efficacy.

OBJECTIVES: Recently, deep learning medical image analysis in orthopedics has become highly active. However, progress has been restricted by the absence of large-scale and standardized ground-truth images. To the best of our knowledge, this study is the first to propose an innovative solution, namely a deep few-shot image augmentation pipeline, that addresses this challenge by synthetically generating knee radiographs for training downstream tasks, with a specific focus on knee osteoarthritis Kellgren-Lawrence (KL) grading. MATERIALS AND METHODS: This study leverages a deep few-shot image augmentation pipeline to generate synthetic knee radiographs. Despite the limited availability of training samples, we demonstrate the capability of our proposed computational strategy to produce high-fidelity plain knee radiographs and use them to successfully train a KL grade classifier. RESULTS: Our experimental results showcase the effectiveness of the proposed computational pipeline. The generated synthetic radiographs exhibit remarkable fidelity, evidenced by the achieved average Frechet Inception Distance (FID) score of 26.33 for KL grading and 22.538 for bilateral knee radiographs. For KL grading classification, the classifier achieved a test Cohen's Kappa and accuracy of 0.451 and 0.727, respectively. Our computational strategy also resulted in a publicly and freely available imaging dataset of 86 000 synthetic knee radiographs. CONCLUSIONS: Our approach demonstrates the capability to produce top-notch synthetic knee radiographs and use them for KL grading classification, even when working with a constrained training dataset. The results obtained emphasize the effectiveness of the pipeline in augmenting datasets for knee osteoarthritis research, opening doors for broader applications in orthopedics, medical image analysis, and AI-powered diagnosis.

A fundamental paradigm shift for in-home sensing is apparent. Special-purpose sensing, where there is a one-to-one relationship between sensors and applications, is evolving into general-purpose sensing, where there is a many-to-many relationship between sensors and applications. This new shift may impact how individuals think about in-home sensing, where utility and comfort are often linked to applications rather than sensed data. We explore the evolution of individuals' perceptions as they become increasingly and contextually aware of sensor capabilities and data characteristics. Through a multi-phase study where 12 participants were progressively led through six exposure conditions across laboratory and home environments, we find that exposure changes represent inflection points for perceptions of utility and comfort with data collection. These changes define opportunities for increasing trust in sensing infrastructures via data- and context-aware interventions, managing over-reliance on awareness notifications, and providing data-enabled "what if" analyses to balance comfort and utility within an individual's unique context and environment.

Low-cost sensors have enabled a wide array of data-driven applications and insights. As a result, encountering spaces with pervasive sensing has become all but unavoidable. This creates a fundamental tension: the success of smart environments will become increasingly dependent on equity of access to data-driven insights and consideration of the privacy expectations of sensed individuals. These concerns highlight the need to bring equity to all stakeholders of smart environments, which in turn would preserve public trust in these smart spaces. In this work, we explored several approaches to identity-obscuring visual representations through a progressive series of experiments. We designed and validated a series of visual representations through stakeholder interactions and tested the ability of these visual representations to limit identification via a crowdsourced study. An evaluation across three months of data gathered within our organization also showed that the identity-obscured data could still be leveraged to accurately count group size. Our contributions lay the groundwork for sensing frameworks that bring utility to all stakeholders of shared spaces while being cognizant of their diverse privacy expectations.

Latency is a pervasive issue in various systems that can significantly impact motor performance and user perception. In medical settings, latency can hinder surgeons' ability to quickly correct movements, resulting in an experience that doesn't align with user expectations and standards of care. Despite numerous studies reporting on the negative effects of latency, there is still a gap in understanding how it impacts the use of augmented reality (AR) in medical settings. This study aims to address this gap by examining how latency impacts motor task performance and subjective perceptions, such as cognitive load, on two display types: a monitor display, traditionally used inside an operating room (OR), and a Microsoft HoloLens 2 display. Our findings indicate that both level of latency and display type impact motor performance, and higher latencies on the HoloLens result in relatively poor performance. However, cognitive load was found to be unrelated to display type or latency, but was dependent on the surgeon's training level. Surgeons did not compromise accuracy to gain more speed and were generally well aware of the latency in the system irrespective of their performance on task. Our study provides valuable insights into acceptable thresholds of latency for AR displays and proposes design implications for the successful implementation and use of AR in surgical settings.

The COVID-19 global pandemic has ignited lightning-fast adoption of digital tools in our communities, organizations, and systems of governance. It also inspired an unprecedented level of providing access to digital devices to communities and individuals lacking prior access. The situation and circumstances provide a unique opportunity to understand digital divides through a new lens. In this work, we contribute a contemporaneous understanding of digital divides beyond access by qualitatively analyzing over 300 calls made to a volunteer-based community IT help desk. We highlight the intertwined network of challenges leading to ecosystem digital divides and contribute new insights into how the complex socio-technical systems of practice, and the tools to support them, must adapt to bridge digital divides more effectively.

Modern operating rooms (OR) are equipped with several ceiling- and wall-mounted screens that display surgical information. These physical displays are restricted in placement, limiting the surgeons’ ability to freely position them in the environment. Our work addresses this issue by exploring the feasibility of using an augmented reality (AR) headset (Microsoft HoloLens 2) as an alternative to traditional surgical screens; leading to a reduced OR footprint and improved surgical ergonomics. We developed several prototypes using state-of-the-art hardware/software and conducted various neurosurgery-related exploratory studies. Initial feedback from users suggests that coloration and resolution of the holographic feed were adequate, however, surgeons frequently commented on tactile/visual asynchrony. This emphasizes the need for novel, more efficient hardware/software solutions to support fine motor tasks in the OR.

Research has shown study habits and skills to be correlated with academic success, calling for a deeper comprehension of these behaviors and processes to design effective interventions for struggling students. Chatbots have recently been used as a persuasive technology to help support behavioral change, making them an intriguing design space for students' study habits and skills. This paper investigated the feasibility of using chatbots for promoting behavioral change of college students majoring in Computer Science (CS). We conducted semi-structured interviews with CS peer-tutors and surveyed university freshmen to understand students' study habits and identify technical intervention opportunities. Inspired by the findings, we designed StudyBuddy, a chatbot prototype deployed in Slack that periodically sends tips, provides assessments of students' study habits via surveys, helps the students break down assignments, recommends academic resources, and sends reminders. We evaluated the usability of the prototype in-depth with 8 students (both first-year and senior students) and 5 course instructors followed by a large scale evaluative survey (n=117) using video of the prototype. Our research identified important design challenges such as building trust and preserving privacy, limiting interaction costs, and supporting both immediate and long-term sustainable support. Likewise, we proposed design recommendations that demonstrate context awareness, personalize the experience based on user preferences, and adapt over time as students mature and grow.

Augmented reality (AR) on smartphone devices allows people to interact with virtually placed objects anchored in the real world through the device’s viewport. Typically, smartphone AR interactions work with the device’s 2D touchscreen disconnected from the modality and depth of the virtual objects. In this paper, we studied 15 participants’ preferences, performances, and cognitive loads on a set of common tasks performed on smartphones at two interaction depths (close-range and distant) with two touchless modalities (hand tracking and screen dwell). We find that distant AR interactions, strongly preferred by the participants, were significantly faster and took less cognitive effort. We observed that within both interaction depths, modalities performed equally. When designing touchless modalities on smartphones, we suggest using distant interactions when overall performance is the top priority, otherwise, using hand tracking or screen dwell as back-ups for each other can be equally effective.

In this paper, we propose a novel solution to optimize the deployment of Radio Frequency (RF) beacons for the purpose of indoor localization. We propose a system that optimizes both the number of beacons and their placement in a given environment. We propose a novel cost-function, called CovBsm, that allows to simultaneously optimize the 3-coverage while maximizing the beacon spreading. Using this cost function, we propose a framework that maximize both the number of beacons and their placement in a given environment. The proposed solution accounts for the indoor infrastructure and its influence on the RF signal propagation by embedding a realistic simulator into the optimization process.

In this paper, we develop a system for the low-cost indoor localization and tracking problem using radio signal strength indicator, Inertial Measurement Unit (IMU), and magnetometer sensors. We develop a novel and simplified probabilistic IMU motion model as the proposal distribution of the sequential Monte-Carlo technique to track the robot trajectory. Our algorithm can globally localize and track a robot with a priori unknown location, given an informative prior map of the Bluetooth Low Energy (BLE) beacons. Also, we formulate the problem as an optimization problem that serves as the Backend of the algorithm mentioned above (Front-end). Thus, by simultaneously solving for the robot trajectory and the map of BLE beacons, we recover a continuous and smooth trajectory of the robot, corrected locations of the BLE beacons, and the time-varying IMU bias. The evaluations achieved using hardware show that through the proposed closed-loop system the localization performance can be improved; furthermore, the system becomes robust to the error in the map of beacons by feeding back the optimized map to the Front-end.

Accurate localization is a fundamental requirement for a variety of applications, ranging from industrial robot operations to location-powered applications on mobile devices. A key technical challenge in achieving this goal is providing a clean and reliable estimation of location from a variety of low-cost, uncalibrated sesnors. Many current techniques rely on Particle Filter (PF) based algorithms. They have proven successful at effectively fusing various sensors inputs to create meaningful location predictions. In this paper we build upon this large corpous of work. Like prior work, our technique fuses Received Signal Strength Indicator (RSSI) measurements from Bluetooth Low Energy (BLE) beacons with map information. A key contribution of our work is a new sensor model for BLE beacons that does not require the mapping from RSSI to distance. We further contribute a novel method of utilizing map information during the initialization of the system and during the resampling phase when new particles are generated. Using our proposed sensor model and map prior information the performance of the overall localization is improved by 1.20 m on comparing the 75thpercentile of the cumulative distribution with traditional localization techniques.

Devices with embedded sensors are permeating the computing landscape, allowing the collection and analysis of rich data about individuals, smart spaces, and their interactions. This class of devices enables a useful array of home automation and connected workplace functionality to individuals within instrumented spaces. Unfortunately, the increasing pervasiveness of sensors can lead to perceptions of privacy loss by their occupants. Given that many instrumented spaces exist as platforms outside of a user's control---e.g., IoT sensors in the home that rely on cloud infrastructure or connected workplaces managed by one's employer---enforcing access controls via a trusted reference monitor may do little to assuage individuals' privacy concerns. This calls for novel enforcement mechanisms for controlling access to sensed data. In this paper, we investigate the interplay between sensor fidelity and individual comfort, with the goal of understanding the design space for effective, yet palatable, sensors for the workplace. In the context of a common space contextualization task, we survey and interview individuals about their comfort with three common sensing modalities: video, audio, and passive infrared. This allows us to explore the extent to which discomfort with sensor platforms is a function of detected states or sensed data. Our findings uncover interesting interplays between content, context, fidelity, history, and privacy. This, in turn, leads to design recommendations regarding how to increase comfort with sensing technologies by revisiting the mechanisms by which user preferences and policies are enforced in situations where the infrastructure itself is not trusted.

Taking breaks from work is an essential and universal practice. In this paper, we extend current research on productivity in the workplace to consider the break habits of knowledge workers and explore opportunities of break logging for personal informatics. We report on three studies. Through a survey of 147 U.S.-based knowledge workers, we investigate what activities respondents consider to be breaks from work, and offer an understanding of the benefit workers desire when they take breaks. We then present results from a two-week in-situ diary study with 28 participants in the U.S. who logged 800 breaks, offering insights into the effect of work breaks on productivity. We finally explore the space of information visualization of work breaks and productivity in a third study. We conclude with a discussion of implications for break recommendation systems, availability and interuptibility research, and the quantified workplace.

We conducted a study that compared basic video conferencing, emergent kinetic video-conferencing techniques, and face-to-face meetings. Remote and co-located participants worked together in groups of three. We show, in agreement with prior literature, the strong adverse impact of being remote on participation levels. We also show that local and remote participants perceived differently their own contributions and others'. Local participants exhibited significantly more overlapping talk with remote participants who used an embodied proxy, than with remote participants in basic-video conferencing (and at a rate similar to overlapping speech for co-located groups). We describe differences in how the technologies were used to follow conversation. Our findings indicate that while the kinetic embodied technology increased local participants' perceived presence of remote teammates, it did not enhance remote participants' own sense of telepresence. We discuss our findings in the context of theories of agency and presence, and discuss how these findings extend our understanding of the promise and limitations of embodied video-conferencing solutions.

Location-enabled applications now permeate the mobile computing landscape. As technologies like Bluetooth Low Energy (BLE) and Apple's iBeacon protocols begin to see widespread adoption, we will no doubt see a proliferation of indoor location enabled application experiences. While not essential to each of these applications, many will require that the location of the device be true and verifiable. In this paper, we present LocAssure, a new framework for trusted indoor location estimation. The system leverages existing technologies like BLE and iBeacons, making the solution practical and compatible with technologies that are already in use today. In this work, we describe our system, situate it within a broad location assurance taxonomy, describe the protocols that enable trusted localization in our system, and provide an analysis of early deployment and use characteristics. Through developer APIs, LocAssure can provide critical security support for a broad range of indoor location applications.

New technologies arise in a number of ways. They may come from advances in scientific research, through new combinations of existing technologies, or by simply imagining what might be possible in the future. This video describes the evolution of Tabletop Telepresence, a system for remote collaboration through desktop videoconferencing combined with a digital desk. Tabletop Telepresence began as a collection of camera, projector, videoconferencing and user interaction technologies. Working together; artists and research scientists combined these technologies into a means of sharing paper documents between remote desktops, interacting with those documents, requesting services (such as translation), and communicating through a videoconference.

In recent years, there has been an explosion of social and collaborative applications that leverage location to provide users novel and engaging experiences. Current location technologies work well outdoors but fare poorly indoors. In this paper we present LoCo, a new framework that can provide highly accurate room-level location using a supervised classification scheme. We provide experiments that show this technique is orders of magnitude more efficient than current state-of-the-art Wi-Fi localization techniques. Low classification overhead and computational footprint make classification practical and efficient even on mobile devices. Our framework has also been designed to be easily deployed and leveraged by developers to help create a new wave of location-driven applications and services.

We present HABLO, an acoustic beacon system for low-range (<; 0.5 m) indoor localization, which remains a difficult task for existing indoor localization technologies. The beacons utilize an audio band which is inaudible to humans but can be received by the microphones on consumer devices such as laptops, tablets and smartphones. HABLO is designed to be low-cost, easy to integrate with existing infrastructure using USB connectors and to be usable with unmodified end-user devices. We present a technical description of the software and hardware implementation of our prototype and show the results of an initial feasibility study of our system for a localization application within an office environment.

Affect influences workplace collaboration and thereby impacts a workplace's productivity. Participants in face-to-face interactions have many cues to each other's affect, but work is increasingly carried out via computer-mediated channels that lack many of these cues. Current presence systems enable users to estimate the availability of other users, but not their affective states or communication preferences. This work demonstrates the feasibility of estimating affective state and communication preferences from a stream of presence states that are already being shared in a deployed presence system.

Faithful sharing of screen contents is an important collaboration feature. Prior systems were designed to operate over constrained networks. They performed poorly even without such bottlenecks. To build a high performance screen sharing system, we empirically analyzed screen contents for a variety of scenarios. We showed that screen updates were sporadic with long periods of inactivity. When active, screens were updated at far higher rates than was supported by earlier systems. The mismatch was pronounced for interactive scenarios. Even during active screen updates, the number of updated pixels were frequently small. We showed that crucial information can be lost if individual updates were merged. When the available system resources could not support high capture rates, we showed ways in which updates can be effectively collapsed. We showed that Zlib lossless compression performed poorly for screen updates. By analyzing the screen pixels, we developed a practical transformation that significantly improved compression rates. Our system captured 240 updates per second while only using 4.6 Mbps for interactive scenarios. Still, while playing movies in fullscreen mode, our approach could not achieve higher capture rates than prior systems; the CPU remains the bottleneck. A system that incorporates our findings is deployed within the lab.

Modern office work practices increasingly breach traditional boundaries of time and place, increasing breakdowns workers encounter when coordinating interactions with colleagues. We conducted interviews with 12 workers and identified key problems introduced by these practices. To address these problems we developed myUnity, a fully functional platform enabling rich workplace awareness and coordination. myUnity is one of the first integrated platforms to span mobile and desktop environments, both in terms of access and sensing. It uses multiple sources to report user location, availability, tasks, and communication channels. A pilot field study of myUnity demonstrated the significant value of pervasive access to workplace awareness and communication facilities, as well as positive behavioral change in day-to-day communication practices for most users. We present resulting insights about the utility of awareness technology in flexible work environments.

Modern office work practices increasingly breach traditional boundaries of time and place, making it difficult to interact with colleagues. To address these problems, we developed myUnity, a software and sensor platform that enables rich workplace awareness and coordination. myUnity is an integrated platform that collects information from a set of independent sensors and external data aggregators to report user location, availability, tasks, and communication channels. myUnity's sensing architecture is component-based, allowing channels of awareness information to be added, updated, or removed at any time. Our current system includes a variety of sensor and data input, including camera-based activity classification, wireless location trilateration, and network activity monitoring. These and other input channels are combined and composited into a single, highlevel presence state. Early studies of a myUnity deployment have demonstrated that use of the platform allows quick access to core awareness information and show its utility in supporting communication and collaboration in the modern workplace.

The use of whiteboards is pervasive across a wide range of work domains. But some of the qualities that make them successful--an intuitive interface, physical working space, and easy erasure--inherently make them poor tools for archival and reuse. If whiteboard content could be made available in times and spaces beyond those supported by the whiteboard alone, how might it be appropriated? We explore this question via ReBoard, a system that automatically captures whiteboard images and makes them accessible through a novel set of user-centered access tools. Through the lens of a seven week workplace field study, we found that by enabling new workflows, ReBoard increased the value of whiteboard content for collaboration.

The modern workplace is inherently collaborative, and this collaboration relies on effective communication among co-workers. Many communication tools -- email, blogs, wikis, Twitter, etc. -- have become increasingly available and accepted in workplace communications. In this paper, we report on a study of communications technologies used over a one year period in a small US corporation. We found that participants used a large number of communication tools for different purposes, and that the introduction of new tools did not impact significantly the use of previously-adopted technologies. Further, we identified distinct classes of users based on patterns of tool use. This work has implications for the design of technology in the evolving ecology of communication tools.

In certain applications such as radiology and imagery analysis, it is important to minimize errors. In this paper we evaluate a structured inspection method that uses eye tracking information as a feedback mechanism to the image inspector. Our two-phase method starts with a free viewing phase during which gaze data is collected. During the next phase, we either segment the image, mask previously seen areas of the image, or combine the two techniques, and repeat the search. We compare the different methods proposed for the second search phase by evaluating the inspection method using true positive and false negative rates, and subjective workload. Results show that gaze-blocked configurations reduced the subjective workload, and that gaze-blocking without segmentation showed the largest increase in true positive identifications and the largest decrease in false negative identifications of previously unseen objects.

We present a new interaction framework for collaborating in multiple display environments (MDEs) and report results from a field study investigating its use in an authentic work setting. Our interaction framework, IMPROMPTU, allows users to share task information across displays via off-the-shelf applications, to jointly interact with information for focused problem solving and to place information on shared displays for discussion and reflection. Our framework also includes a lightweight interface for performing these and related actions. A three week field study of our framework was conducted in the domain of face-to-face group software development. Results show that teams utilized almost every feature of the framework in support of a wide range of development-related activities. The framework was used most to facilitate opportunistic collaboration involving task information. Teams reported wanting to continue using the framework as they found value in it overall.

Software developers spend significant time gaining and maintaining awareness of fellow developers' activities. FASTDash is a new interactive visualization that seeks to improve team activity awareness using a spatial representation of the shared code base that highlights team members' current activities. With FASTDash, a developer can quickly determine which team members have source files checked out, which files are being viewed, and what methods and classes are currently being changed. The visualization can be annotated, allowing programmers to supplement activity information with additional status details. It provides immediate awareness of potential conflict situations, such as two programmers editing the same source file. FASTDash was developed through user-centered design, including surveys, team interviews, and in situ observation. Results from a field study show that FASTDash improved team awareness, reduced reliance on shared artifacts, and increased project-related communication. Additionally, the team that participated in our field study continues to use FASTDash.

Productive collaboration in a multiple-device environment (MDE) requires an effective interface for efficiently managing applications among devices. Though many interfaces exist, there is little empirical understanding of how they affect collaboration. This paper reports results from a user study comparing how well three classes of interfaces; textual, map, and iconic, support application management during realistic, collaborative activities in an MDE. From empirical results, observations, and an analysis of how users interacted with each interface, we produced a set of design lessons for improving management interfaces. The lessons were demonstrated within the iconic interface, but they are just as applicable to other interfaces. This work contributes further understanding of how to design effective management interfaces for MDEs.

Iconic interfaces offer a promising interaction metaphor for effectively managing applications in multi-device environments. However, current implementations scale poorly for even a modest number of applications and do not allow users to maintain adequate awareness of the workspace. To overcome these limitations, we have designed new interaction techniques and prototyped them within a new iconic interface. Our interface uses zooming and animation-based interactions to improve scalability and uses application icons and portal views with real-time updates to enhance awareness. Results from a user study confirm the efficacy of these techniques. These techniques can be used to improve the broader class of iconic and portal-based interfaces.