Summary form only given. What will be the next computing paradigm as the ubiquitous computing wave crests: autonomobiles, robots, virtual reality, internet of things, intelligent agents? While pundits search for the next big thing among a dizzying array of shiny ideas, the truth is that pervasive technologies have reached critical mass that we no longer need to ask "what if" and we can shift our attention to "what when". One picture of that future is a recasting of how we think about designing digital experiences - rather than systems that react to user direction, we can design systems that respond dynamically to the users' attention, engagement and context: Responsive Media. The same machine learning technologies that have made speech and image recognition surprisingly accurate are also enhancing our devices' abilities to sense user activities, emotions and intentions and to deliver services and information proactively. Media experiences will be dramatically changed by the next generation of these technologies embedded into smartphones and VR goggles and robots and smart homes and autonomobiles so that they not only sense the audience's engagement in real time, but they can also predict disengagement and prevent it by dynamically shifting the content to appeal to an individual's preferences, emotion state and situation. More than just media experiences, imagine robots that can sense a child's frustration and actively assist in the homework, digital assistants that do not interrupt inappropriately, semi-autonomobiles that ensure the media is not disrupting the driver's attention demands, and more. Responsive media will be more like an engaging conversation among humans, rather than just passive consumption. What are the requirements for a conversational interaction? This talk will paint a picture and challenge the audience to identify the remaining technology barriers, architectures, business ecosystems, threats, and yes, the killer applications. I seek your input as we create the future beyond ubiquitous computing.

We present SurfaceLink, a system where users can make natural surface gestures to control association and information transfer among a set of devices that are placed on a mutually shared surface (e.g., a table). SurfaceLink uses a combination of on-device accelerometers, vibration motors, speakers and microphones (and, optionally, an off-device contact microphone for greater sensitivity) to sense gestures performed on the shared surface. In a controlled evaluation with 10 participants, SurfaceLink detected the presence of devices on the same surface with 97.7% accuracy, their relative arrangement with 89.4% accuracy, and various single- and multi-touch surface gestures with an average accuracy of 90.3%. A usability analysis showed that SurfaceLink has advantages over current multi-device interaction techniques in a number of situations.

The routines of information work are commonplace yet difficult to characterize. Although cognitive models have successfully characterized routine tasks within which there is little variation, a large body of ethnomethodological research has identified the inherent nonroutineness of routines in information work. We argue that work does not fall into discrete classes of routine versus nonroutine; rather, task performance lies on a continuum of routineness, and routineness metrics are important to the understanding of workplace multitasking. In a study of 10 information workers shadowed for 3 whole working days each, we utilize the construct of working sphere to model projects/tasks as a network of humans and artifacts. Employing a statistical technique called T-pattern analysis, we derive measures of routineness from these real-world data. In terms of routineness, we show that information workers experience archetypes of working spheres. The results indicate that T-patterns of interactions with information and computational media are important indicators of facets of routineness and that these measures are correlated with workers' affective states. Our results are some of the first to demonstrate how regular temporal patterns of media interaction in tasks are related to stress. These results suggest that designs of systems to facilitate so-called routine work should consider the degrees to which a person's working spheres fall along varying facets of routineness.

We examine opportunities for ubiquitous technologies in retail shopping, jewelry shopping in this case, to supplement the unique information needs inherent to physical trials of tactile products. We describe an iterative design approach to develop a mirror system that records and matches images across jewelry trials called the Countertop Responsive Mirror. The key technological distinction of our system from prior technologies is the use of "matched access," which automatically retrieves images that match a scene shown in separately accessed images. This not only helps shoppers compare jewelry but also promotes interactions among all parties during shopping. We report qualitative findings from multiple field trials of the system. This paper contributes to a body of research on the design and introduction of new technologies into retail shopping that provide value to all users without disruption to their normative practices and behaviors.

This paper introduces a new representation for describing routine tasks, called temporal task footprints. Routines are characterized by their temporal regularity or rhythm. Temporal pattern analysis (T-patterns) can be used to isolate frequent recurrent patterns in routine tasks that appear repeatedly in the same temporal configuration. Using tf-idf statistics, each task can then be defined in terms of its temporal task footprint, a ranked list of temporal patterns along with their typical frequencies. Experimental evaluations using data of 29 days observing and logging 10 subjects showed that temporal task footprints of application windows, email and document usage outperform decision tree and SVMs in recognizing the subjects' tasks.

This paper presents a context-aware mobile recommender system, codenamed Magitti. Magitti is unique in that it infers user activity from context and patterns of user behavior and, without its user having to issue a query, automatically generates recommendations for content matching. Extensive field studies of leisure time practices in an urban setting (Tokyo) motivated the idea, shaped the details of its design and provided data describing typical behavior patterns. The paper describes the fieldwork, user interface, system components and functionality, and an evaluation of the Magitti prototype.

In this paper, we describe the architecture of the vision system for the Responsive Mirror, a novel system for retail fitting rooms that enables online social fashion comparisons in physical stores based on multi-camera perception. This vision system provides implicitly controlled real-time interaction for “self ” and “social ” clothing comparisons by automatically tracking user’s motion as she tries on clothes. We describe the key components of the motion-tracking and clothes-recognition systems and evaluate their effectiveness against images collected during a previous user study and a dataset of images representing content from a social fashion network. Author Keywords Ubiquitous computing, computer vision, multi-camera,

One of the main challenges in building multi-user mobile information systems for real-world deployment lies in the development of scalable systems. Recent work on scaling infrastructure for conventional web services using distributed approaches can be applied to the mobile space, but limitations inherent to mobile devices (computational power, battery life) and their communication infrastructure (availability and quality of network connectivity) challenge system designers to carefully design and optimize their software architectures. Additionally, notions of mobility and position in space, unique to mobile systems, provide interesting directions for the segmentation and scalability of mobile information systems. In this paper we describe the implementation of a mobile recommender system for leisure activities, codenamed Magitti, which was built for commercial deployment under stringent scalability requirements. We present concrete solutions addressing these scalability challenges, with the goal of informing the design of future mobile multi-user systems.

Efficient coordination of collaboration requires sharing information about collaborators' current and future availability. We describe the usage of an awareness system called Awarenex that shared real-time awareness information to help coordinate activities at the current moment. We also developed a prototype called Lilsys that used sensors to gather additional awareness information that would help avoid disruptions when users are currently unavailable for interaction. Our experiences over time in designing and using prototypes that share awareness cues for current availability led us to identify temporal patterns that could help predict future reachability. Rhythm awareness is having a sense of regularly recurring temporal patterns that can help coordinate interactions among collaborators. Rhythm awareness is difficult to establish within distributed groups that are separated by distance and time zone. We describe rhythmic temporal patterns observed in activity data collected from users of the Awarenex prototype. Analyzing logs of Awarenex usage over time enabled us to construct a computational model of temporal patterns. We explored how to apply those patterns and model to predict future reachability among distributed team members. We discuss trade-offs in the design of collaborative applications that rely on human- and machine-interpretation of rhythm awareness cues. We also conducted a design study that elicited reactions to a variety of end-user visualizations of rhythmic patterns and investigated how well our computational model characterized their everyday routines.

As communications systems increasingly gather and propagate information about people's reachability or ``presence'', users need better tools to minimize undesired interruptions while allowing desired ones. We review the salient elements of presence and availability that people use when initiating face-to-face communication. We discuss problems with current strategies for managing one's availability in telecommunication media. We describe a prototype system called Lilsys which passively collects availability cues gathered from users' actions and environment using ambient sensors and provides machine inferencing of unavailability. We discuss observations and design implications from deploying Lilsys.

People use their awareness of others' temporal patterns to plan work activities and communication. This paper presents algorithms for programatically detecting and modeling temporal patterns from a record of online presence data. We describe analytic and end-user visualizations of rhythmic patterns and the tradeoffs between them. We conducted a design study that explored the accuracy of the derived rhythm models compared to user perceptions, user preference among the visualization alternatives, and users' privacy preferences. We also present a prototype application based on the rhythm model that detects when a person is "away" for an extended period and predicts their return. We discuss the implications of this technology on the design of computer-mediated communication.

We examined records of minute-by-minute computer activity coupled with information about the location of the activity, online calendar appointments, and e-mail activity. We present a number of visualizations of the data that exhibit meaningful patterns in users' activities. We demonstrate how the patterns vary between individuals and within individuals according to time of day, location, and day of the week. Some patterns augment the schedule information found in people's online calendars. We discuss applications for group coordination (especially across time zones) plus opportunities for future research. In light of the popularity of instant messaging, this research identifies some of the benefits and privacy risks associated with the uses of online presence and awareness information.

We explored the use of awareness information to facilitate communication by developing a series of prototypes. The ConNexus prototype integrates awareness information, instant messaging, and other communication channels in an interface that runs on a desktop computer. The Awarenex prototype extends that functionality to wireless handheld devices, such as a Palm. A speech interface also enables callers to make use of the awareness information over the telephone. While the prototypes offer similar functionality, the interfaces reflect the different design affordances and use context of each platform. We discuss the design implications of providing awareness information on devices with varying interface and network characteristics.

We describe problems associated with accessing data resources external to the application, which we term externalities, in replicated synchronous collaborative applications. Accessing externalities such as files, databases, network connections, environment variables, and the system clock is not as straightforward in replicated collaborative software as in single-user applications or centralized collaborative systems. We describe ad hoc solutions that add to development cost and complexity because the developer must program different behavior for different replicas. We introduce a novel general approach to accessing externalities uniformly in a replicated collaborative system. The approach uses a semireplicated architecture where the actual externality resides at a single location and is accessed via replicated proxies. This approach allows developers of replicated synchronous groupware to (1) use similar externality access mechanisms as in traditional single-user applications, and (2) program all replicas to execute the same behavior. We describe a general design for proxied access to read-only, write-only, and read-write externalities and discuss the tradeoffs of this semireplicated approach over full, literal replication and the class of applications to which this approach can be successfully applied. We also describe details of a prototype implementation of this approach within a replicated collaboration-transparency system, called Flexible JAMM (Java Applets Made Multiuser).

No abstract available.

This article presents a critique of conventional collaboration transparency systems, also called “application-sharing” systems, which provide the real-time shared use of legacy single-user applications. We find that conventional collaboration transparency systems are inefficient in their use of network resources and lack support for key groupware principles: concurrent work, relaxed WYSIWIS, and group awareness. Next, we present an alternative approach to implementing collaboration transparency that provides many features previously seen only in collaboration-aware applicaitons. Our approach is based on a replicated architecture where selected single-user interface components are dynamically replaced by multiuser versions. The replacement occurs at run-time and is transparent to the single-user application and its developers.. As an instance of this approach, we describe its incorporation into a Java-based collaboration transparency system for serializable, Swing-based Java applications, called Flexible JAMM (Java Applets Made Multiuser). To validate that the flexible collaboration transparency system is truly an improvement over conventional systems, we conducted an empirical study of collaborators performing both tightly and loosely coupled tasks using Flexible JAMM versus a representative conventional collaboration transparency system, Microsoft NetMeeting. Completion times were significantly faster in the loosely coupled task using Flexible JAMM and were not adversely affected in the tightly coupled task. Accuracy was equivalent for both systems. Participants greatly preferred Flexible JAMM.

Conventional collaboration-transparency systems, which provide real-time shared use of legacy single-user applications, are inefficient in their use of network resources and lack support for key groupware principles: concurrent work, relaxed WYSIWIS, and group awareness. We present an alternative implementation approach to collaboration transparency that provides many features previously seen only in collaboration-aware applications. Our approach is based on an object-oriented replicated architecture where selected single-user interface objects are dynamically replaced by multi-user extensions. The replacement occurs at run-time and is transparent to the single-user application and its developers. As an instance of this approach, we describe its incorporation into a new Java-based collaboration-transparency system for serializable, Swing-based Java applications, called Flexible JAMM (Java Applets Made Multiuser). We conducted an empirical study to evaluate the effectiveness of Flexible JA...

People interact together in all aspects of life and, as computers have become prevalent, users seek computer support for their interactions.The WWW provides an unprecedented opportunity for users to interact with each other, and the advent of JavaThfl has created a consistent computing environment to support synchronous collaboration.We describe JAMM, a prototype Java runtime environment that supports the shared use of existing Java applets, thus leveraging the existing base of software for synchronous collaboration.Our approach is based on a replicated architecture, where each user maintains their own copy of the Java applet, and the users' input events are broadcast to each applet copy.We discuss solutions to certain key problems, such as unanticipated sharing, supporting late-joiners, and replicating input sources other than user inputs (e.g., files, sockets, and random number generators).