In this special issue ofIEEE Transactions on Visualization and Computer Graphics (TVCG), we are pleased to present theTVCGpapers from the 17th IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2018), held October 16–20 in Munich, Germany. ISMAR continues the 20-year long tradition of IWAR, ISMR, and ISAR, and is undoubtedly the premier conference for mixed and augmented reality in the world.

GPUs have recently enjoyed increased popularity as general purpose software accelerators in multiple application domains including computer vision and natural language processing. However, there has been little exploration into the performance and energy trade-offs mobile GPUs can deliver for the increasingly popular workload of deep-inference audio sensing tasks, such as, spoken keyword spotting in energy-constrained smartphones and wearables. In this paper, we study these trade-offs and introduce an optimization engine that leverages a series of structural and memory access optimization techniques that allow audio algorithm performance to be automatically tuned as a function of GPU device specifications and model semantics. We find that parameter optimized audio routines obtain inferences an order of magnitude faster than sequential CPU implementations, and up to 6.5x times faster than cloud offloading with good connectivity, while critically consuming 3-4x less energy than the CPU. Under our optimized GPU, conventional wisdom about how to use the cloud and low power chips is broken. Unless the network has a throughput of at least 20Mbps (and a RTT of 25 ms or less), with only about 10 to 20 seconds of buffering audio data for batched execution, the optimized GPU audio sensing apps begin to consume less energy than cloud offloading. Under such conditions we find the optimized GPU can provide energy benefits comparable to low-power reference DSP implementations with some preliminary level of optimization; in addition to the GPU always winning with lower latency.

Virtual reality head-mounted displays (VR HMDs) are attracting users with the promise of full sensory immersion in virtual environments. Creating the illusion of immersion for a near-eye display results in very heavy rendering workloads: low latency, high framerate, and high visual quality are all needed. Tethered VR setups in which the HMD is bound to a powerful gaming desktop limit mobility and exploration, and are difficult to deploy widely. Products such as Google Cardboard and Samsung Gear VR purport to offer any user a mobile VR experience, but their GPUs are too power-constrained to produce an acceptable framerate and latency, even for scenes of modest visual quality.

Abstract VR headsets and hand‐held devices are not powerful enough to render complex scenes in real‐time. A server can take on the rendering task, but network latency prohibits a good user experience. We present a new image‐based rendering (IBR) architecture for masking the latency. It runs in real‐time even on very weak mobile devices, supports modern game engine graphics, and maintains high visual quality even for large view displacements. We propose a novel server‐sidedual‐viewrepresentation that leverages an optimally‐placed extra view and depth peeling to provide the client with coverage for filling disocclusion holes. This representation is directly rendered in a novel wide‐angle projection with favorable directional parameterization. A new client‐side IBR algorithm uses a pre‐transmitted level‐of‐detail proxy with an encaging simplification and depth‐carving to maintain highly complex geometric detail. We demonstrate our approach with typical VR / mobile gaming applications running on mobile hardware. Our technique compares favorably to competing approaches according to perceptual and numerical comparisons.

In cloud gaming, servers perform remote rendering on behalf of thin clients. Such a server must deliver sufficient frame rate (at least 30fps) to each of its clients. At the same time, each client desires an immersive experience, and therefore the server should also provide the best graphics quality possible to each client. Statically provisioning time slices of the server GPU for each client suffers from severe underutilization because clients can come and go, and scenes that the clients need rendered can vary greatly in terms of GPU resource usage over time.

Even though mobile devices are ubiquitous, the conceptually simple endeavor of using co-located devices for multi-user experiences is cumbersome. It may not even be possible when certain apps are not widely available.

Social networks are the most engaging applications on mobile devices, and they are becoming the main sources for users to consume content. However, content retrieval, especially for embedded links and multimedia, can often be too slow, too energy hungry or too expensive for on-the-go mobile users. To address these issues, we collect and analyze a large set of traces from over 6000 real-life users of a popular mobile Twitter client. Based on the unique challenges identified from our dataset, we present inference-based social network content prefetcher, Earlybird. It uses the specific signals unique to social data in order to retrieve news feeds and associated links and multimedia ahead of users' usage. Our regression-based content prediction model is able to estimate a user's likely content interests 55% of the time. Second, we develop a prefetch scheduling scheme to maximize delay reduction under users' resource constraints. For validation, we apply Earlybird to our collected dataset. We show that on average users can reduce their delays by 62% at the cost of no more than 3% battery and 40MB/month cellular data.

No abstract available.

Gaming on phones, tablets and laptops is very popular. Cloud gaming - where remote servers perform game execution and rendering on behalf of thin clients that simply send input and display output frames - promises any device the ability to play any game any time. Unfortunately, the reality is that wide-area network latencies are often prohibitive; cellular, Wi-Fi and even wired residential end host round trip times (RTTs) can exceed 100ms, a threshold above which many gamers tend to deem responsiveness unacceptable.

This paper presents a recognition scheme for fine-grain gestures. The scheme leverages directional antenna and short-range wireless propagation properties to recognize a vocabulary of action-oriented gestures from the American Sign Language. Since the scheme only relies on commonly available wireless features such as Received Signal Strength (RSS), signal phase differences, and frequency subband selection, it is readily deployable on commercial-off-the-shelf IEEE 802.11 devices. We have implemented the proposed scheme and evaluated it in two potential application scenarios: gesture-based electronic activation from wheelchair and gesture-based control of car infotainment system. The results show that the proposed scheme can correctly identify and classify up to 25 fine-grain gestures with an average accuracy of 92% for the first application scenario and 84% for the second scenario.

This paper investigates the development of accurate and efficient classifiers to identify misbehaving users (i.e., “flashers”) in a mobile video chat application. Our analysis is based on video session data collected from a mobile client that we built that connects to a popular random video chat ser

No abstract available.

Mobile phones have evolved from communication devices to indispensable accessories with access to real-time content. The increasing reliance on dynamic content comes at the cost of increased latency to pull the content from the Internet before the user can start using it. While prior work has explored parts of this problem, they ignore the bandwidth costs of prefetching, incur significant training overhead, need several sensors to be turned on, and do not consider practical systems issues that arise from the limited background processing capability supported by mobile operating systems. In this paper, we make app prefetch practical on mobile phones. Our contributions are two-fold. First, we design an app prediction algorithm, APPM, that requires no prior training, adapts to usage dynamics, predicts not only which app will be used next but also when it will be used, and provides high accuracy without requiring additional sensor context. Second, we perform parallel prefetch on screen unlock, a mechanism that leverages the benefits of prediction while operating within the constraints of mobile operating systems. Our experiments are conducted on long-term traces, live deployments on the Android Play Market, and user studies, and show that we outperform prior approaches to predicting app usage, while also providing practical ways to prefetch application content on mobile phones.

Online video chat services such as Chatroulette and Omegle randomly match users in video chat sessions and have become increasingly popular, with tens of thousands of users online at anytime during a day. Our interest is in examining user behavior in the growing domain of mobile video, and in particular how users behave in such video chat services as they are extended onto mobile clients. To date, over four thousand people have downloaded and used our Android-based mobile client, which was developed to be compatible with an existing video chat service. The paper provides a first-ever detailed large scale study of mobile user behavior in a random video chat service over a three week period. This study identifies major characteristics such as mobile user session durations, time of use, demographic distribution and the large number of brief sessions that users click through to find good matches. Through content analysis of video and audio, as well as analysis of texting and clicking behavior, we discover key correlations among these characteristics, e.g., normal mobile users are highly correlated with using the front camera and with the presence of a face, whereas misbehaving mobile users have a high negative correlation with the presence of a face.

Mobile gaming is a big driver of app marketplaces. However, few mobile games deliver truly distinctive gameplay experiences for ad hoc collocated users. As an example of such an experience, consider a sword fight dual between two users facing each other where each user's phone simulates a sword. With phone in hand, the users' thrusts and blocks translate to attacks and counterattacks in the game. Such Phone-to-Phone Mobile Motion Games (MMG) represent interesting and novel gameplay for ad hoc users in the same location. One enabler for an MMG game like sword fight is continuous, accurate distance ranging. Existing ranging schemes cannot meet the stringent requirements of MMG games: speed, accuracy, and noise robustness. In this work, we design FAR, a new ranging scheme that can localize at 12 Hz with 2-cm median error while withstanding up to 0-dB noise, multipath, and Doppler effect issues. Our implementation runs on commodity smartphones and does not require any external infrastructure. Moreover, distance measurement accuracy is comparable to that of Kinect, a fixed-infrastructure motion capture system. Evaluation on users playing two prototype games indicate that FAR can fully support dynamic game motion in real time.

A growing number of mobile apps are exploiting smartphone sensors to infer user behavior, activity, or context. Inference requires training using labeled ground truth data. Obtaining labeled data for new apps is a "chicken-egg" problem. Without a reasonable amount of labeled data, apps cannot provide any service. But until an app provides useful service it is not worth installing and has no opportunity to collect user data. This paper aims to address this problem. Our intuition is that even though users are different, they exhibit similar patterns on certain sensing dimensions. For instance, different users may walk and drive at different speeds, but certain speeds will indicate driving for all users. These common patterns could be used as "seeds" to model new users through semi-supervised learning. We prototype a technique to automatically extract the commonalities to seed personalized inference models for new users. We evaluate the proposed technique through example apps and real world data.

Many context aware applications can benefit from using high-level sensing results with semantic meanings (e.g, busy/idle). This paper proposes a platform design that provides high-level "virtual sensor" abstractions and enables new virtual sensors to be bootstrapped from existing ones.

Online video chat services such as Chatroulette [1] and Omegle [2] that randomly match pairs of users in video chat sessions have become increasingly popular, with over twenty thousand online users at anytime during a day. A key problem encountered in such systems is the presence of misbehaving users ("flashers") and obscene content. Our previous works [3] [4] prove that using some image recognition methods (skin-detection, dense SIFT) and machine learning algorithms could achieve significantly higher recall and better precision for flasher detection. Nowadays, with the rapid development of advanced mobile phones with both front and back cameras, we expect mobile video chat to become a popular extension of online video chat services. However, because of the computation-intensive features used by our previous solutions and mobile phones' hardware limitations such as memory size and CPU capacity, it is difficult to directly apply our previous works to mobile platforms. As smartphones are increasingly equipped with diverse sensing capabilities, we plan to utilize this multi-dimensional sensor information to extend flasher detection on mobile platform. This project explores how we can mine accelerometer and other mobile sensor data to infer some clues to optimize flasher detection accuracy while reducing the computation demands of flasher detection on the mobile device.

As mobile apps become more closely integrated into our everyday lives, mobile app interactions ought to be rapid and responsive. Unfortunately, even the basic primitive of launching a mobile app is sorrowfully sluggish: 20 seconds of delay is not uncommon even for very popular apps.

Mobile gaming is a big driver of app marketplaces. However, few mobile games deliver truly distinctive gameplay experiences for ad hoc collocated users. As an example of such an experience, consider a sword fight dual between two users facing each other where each user's phone simulates a sword. With phone in hand, the users' thrusts and blocks translate to attacks and counterattacks in the game. Such Phone-to-Phone Mobile Motion Games (MMG) represent interesting and novel gameplay for ad hoc users in the same location. One enabler for an MMG game like sword fight is continuous, accurate distance ranging. Existing ranging schemes cannot meet the stringent requirements of MMG games: speed, accuracy and noise robustness. In this work, we design FAR, a new ranging scheme that can localize at 12Hz with 2cm median error while withstanding up to 0dB noise, multipath and Doppler effect issues. Our implementation runs on commodity smartphones and does not require any external infrastructure. Moreover, distance measurement accuracy is comparable to that of Kinect, a fixed-infrastructure motion capture system. Evaluation on users playing two prototype games indicate that FAR can fully support dynamic game motion in real-time.

The state of the art rigid object recognition algorithms are based on the bag of words model, which represents each image in the database as a sparse vector of visual words. We propose a new algorithm to select informative features from images in the database. which can save the memory cost when the database is large and reduce the length of the inverted index so it can improve the recognition speed. Experiments show that only using the informative features selected by our algorithm has better recognition performance than the previous methods.

Sensor convergence on the mobile phone is spawning a broad base of new and interesting mobile applications. As applications grow in sophistication, raw sensor readings often require classification into more useful application-specific high-level data. For example, GPS readings can be classified as running, walking or biking. Unfortunately, traditional classifiers are not built for the challenges of mobile systems: energy, latency, and the dynamics of mobile.

High-speed, locational, phone-to-phone (HLPP) games and apps constitute a provocative class of mobile apps that are currently unsupported on commodity mobile devices. This work looks at a key problem for enabling HLPP: a specific variant of the localization problem in which two phones estimate each other's relative positions in 3D space without infrastructure support. Moreover, position estimates should reflect changes due to the phones' possible mobility.

We present a demonstration of a phone-to-phone Sword Fight! game. It utilizes our solution for achieving high speed 3D continuous localization described in the accompanying conference paper [1]. The approach uses acoustic cues based on time-difference of arrival and power level. It assumes at least two microphones and one speaker per phone, which is common on new smartphones. Accelerometers and digital compasses assist in resolving ambiguous acoustic-only localization. Continuous localization is achieved with the aid of a loose time synchronization protocol and a Kalman filter. Lastly, practical gameplay issues are addressed.

This work extends the declarative sensornet programming model with automated program optimizations that attempt to minimize energy expenditure at various points in the communication stack.

Declarative languages have recently been proposed for many new applications outside of traditional data management. Since these are relatively early research efforts, it is important that the architectures of these declarative systems be extensible, in order to accommodate unforeseen needs in these new domains. In this paper, we apply the lessons of declarative systems to the internals of a declarative engine. Specifically, we describe our design and implementation of Evita Raced , an extensible compiler for the OverLog language used in our declarative networking system, P2. Evita Raced is a metacompiler : an OverLog compiler written in OverLog. We describe the minimalist architecture of Evita Raced, including its extensibility interfaces and its reuse of P2's data model and runtime engine. We demonstrate that a declarative language like OverLog is well-suited to expressing traditional and novel query optimizations as well as other query manipulations, in a compact and natural fashion. Finally, we present initial results of Evita Raced extended with various optimization programs, running on both Internet overlay networks and wireless sensor networks.

Large-scale sensor network applications require in-network processing and data fusion to compute statistically relevant summaries of the sensed measurements. This paper studies distributed message-passing algorithms, in which neighboring nodes in the network pass local information relevant to a global computation, for performing statistical inference. We focus on the class of reweighted belief propagation (RBP) algorithms, which includes as special cases the standard sum-product and max-product algorithms for general networks with cycles, but in contrast to standard algorithms has attractive theoretical properties (uniqueness of fixed points, convergence, and robustness). Our main contribution is to design and implement a practical and modular architecture for implementing RBP algorithms in real networks. In addition, we show how intelligent scheduling of RBP messages can be used to minimize communication between motes and prolong the lifetime of the network. Our simulation and Mica2 mote deployment indicate that the proposed algorithms achieve accurate results despite real-world problems such as dying motes, dead and asymmetric links, and dropped messages. Overall, the class of RBP provides provides an ideal fit for sensor networks due to their distributed nature, requiring only local knowledge and coordination, and little requirements on other services such as reliable transmission.

Sensor networks are notoriously difficult to program, given that they encompass the complexities of both distributed and embedded systems. To address this problem, we present the design and implementation of a declarative sensor network platform, DSN: a declarative language, compiler and runtime suitable for programming a broad range of sensornet applications. We demonstrate that our approach is a natural fit for sensor networks by specifying several very different classes of traditional sensor network protocols, services and applications entirely declaratively -- these include tree and geographic routing, link estimation, data collection, event tracking, version coherency, and localization. To our knowledge, this is the first time these disparate sensornet tasks have been addressed by a single high-level programming environment. Moreover, the declarative approach accommodates the desire for architectural flexibility and simple management of limited resources. Our results suggest that the declarative approach is well-suited to sensor networks, and that it can produce concise and flexible code by focusing on what the code is doing, and not on how it is doing it.

Wireless sensor networks are proving to be useful in a variety of settings. A core challenge in these networks is to minimize energy consumption. Prior database research has proposed to achieve this by pushing data-reducing operators like aggregation and selection down into the network. This approach has proven unpopular with early adopters of sensor network technology, who typically want to extract complete "dumps" of the sensor readings, i.e., to run "SELECT *" queries. Unfortunately, because these queries do no data reduction, they consume significant energy in current sensornet query processors. In this paper we attack the "SELECT " problem for sensor networks. We propose a robust approximate technique called Ken that uses replicated dynamic probabilistic models to minimize communication from sensor nodes to the network’s PC base station. In addition to data collection, we show that Ken is well suited to anomaly- and event-detection applications. A key challenge in this work is to intelligently exploit spatial correlations across sensor nodes without imposing undue sensor-to-sensor communication burdens to maintain the models. Using traces from two real-world sensor network deployments, we demonstrate that relatively simple models can provide significant communication (and hence energy) savings without undue sacrifice in result quality or frequency. Choosing optimally among even our simple models is NPhard, but our experiments show that a greedy heuristic performs nearly as well as an exhaustive algorithm.

Sensor network applications tend to exhibit significant high-level commonalities along several major dimensions that have heretofore been underexposed, particularly in the areas of collection and dissemination. We have developed a component library, sdlib, which presents the fundamental abstractions of collection and dissemination as part of a dataflow sytem. This allows application developers to rapidly develop applications at the nesC level. This means that sdlib maintains significant expressivity while operating efficiently.We have built four applications, each faithful to a mature monolithic application, on top of sdlib to compare its performance to that of original. We find that applications implemented with sdlib are much simpler to write, just as resource efficient, and perform comparably to monolithic implementations.

A basic task in sensor networks is to interactively gather data from a subset of the sensor nodes. When data needs to be gathered from a selected set of nodes in the network, existing communication schemes often behave poorly. In this paper, we study the algorithmic challenges in efficiently routing a fixed-size packet through a small number of nodes in a sensor network, picking up data as the query is routed. We show that computing the optimal routing scheme to visit a specific set of nodes is NP-complete, but we develop approximation algorithms that produce plans with costs within a constant factor of the optimum. We enhance the robustness of our initial approach to accommodate the practical issues of limited-sized packets as well as network link and node failures, and examine how different approaches behave with dynamic changes in the network topology. Our theoretical results are validated via an implementation of our algorithms on the TinyOS platform and a controlled simulation study using Matlab and TOSSIM.

The compact, mobile device, such as the PocketPC, is often regarded as little more than a handy, smaller "portal" by which to access information kept on larger back-end machines such as exchange servers and WWW servers. However, the true value of the device may ultimately result from more sophisticated approaches by which the device can fully recognize and change its behavior based on its context, such as location, remaining battery life, available networking, and user preferences. We present "UVa Bus.NET", a testbed at the University of Virginia for developing and evaluating general, context-aware, mobile solutions. We use .NET, .NET Compact Framework, GPS-enabled devices, and wireless networking inside buildings to notify students and professors of impending appointments and class meetings, give directions to their next appointment, and even direct them to the real-time location of the most appropriate bus to catch. The longer-term goals of UVa Bus.NET are also presented: to provide a more predictable experience to mobile device users by hiding and otherwise managing resource limitations.