The computational capabilities of recent mobile devices enable the processing of natural features for Augmented Reality (AR), but the scalability is still limited by the devices' computation power and available resources. In this paper, we propose EdgeXAR, a mobile AR framework that utilizes the advantages of edge computing through task offloading to support flexible camera-based AR interaction. We propose a hybrid tracking system for mobile devices that provides lightweight tracking with 6 Degrees of Freedom and hides the offloading latency from users' perception. A practical, reliable and unreliable communication mechanism is used to achieve fast response and consistency of crucial information. We also propose a multi-object image retrieval pipeline that executes fast and accurate image recognition tasks on the cloud and edge servers. Extensive experiments are carried out to evaluate the performance of EdgeXAR by building mobile AR apps upon it. Regarding the Quality of Experience (QoE), the mobile AR apps powered by EdgeXAR framework run on average at the speed of 30 frames per second with precise tracking of only 1-2 pixel errors and accurate image recognition of at least 97% accuracy. As compared to Vuforia, one of the leading commercial AR frameworks, EdgeXAR transmits 87% less data while providing a stable 30FPS performance and reducing the offloading latency by 50 to 70% depending on the transmission medium. Our work facilitates the large-scale deployment of AR as the next generation of ubiquitous interfaces.

Throughout the past decade, numerous interaction techniques have been designed for mobile and wearable devices. Among these devices, smartglasses mostly rely on hardware interfaces such as touchpad and buttons, which are often cumbersome and counterintuitive to use. Furthermore, smartglasses feature cheap and low-power hardware preventing the use of advanced pointing techniques. To overcome these issues, we introduce UbiPoint, a freehand mid-air interaction technique. UbiPoint uses the monocular camera embedded in smartglasses to detect the user's hand without relying on gloves, markers, or sensors, enabling intuitive and non-intrusive interaction. We introduce a computationally fast and light-weight algorithm for fingertip detection, which is especially suited for the limited hardware specifications and the short battery life of smartglasses. UbiPoint processes pictures at a rate of 20 frames per second with high detection accuracy - no more than 6 pixels deviation. Our evaluation shows that UbiPoint, as a mid-air non-intrusive interface, delivers a better experience for users and smart glasses interactions, with users completing typical tasks 1.82 times faster than when using the original hardware.

Smartglasses mostly rely on hardware interfaces such as touch-pad and buttons, which are often cumbersome and counter-intuitive to use. Furthermore, smartglasses feature cheap and low-power hardware preventing the use of advanced pointing techniques. To overcome these issues, we introduce TiPoint, a freehand mid-air interaction technique. TiPoint uses the monocular camera embedded in smartglasses to detect the user's hand, enabling intuitive and non-intrusive interaction. We introduce a light-weight algorithm for fingertip detection, which is especially suited for the limited hardware specifications and the short battery life time of smartglasses. Our evaluation shows that TiPoint as a mid-air non-intrusive interface delivers a better experience for users and smart glasses interactions, with users completing typical tasks 1.82 times faster than when using the original hardware.

Most smart glasses have a small and limited field of view. The head-mounted display often spreads between the human central and peripheral vision. In this paper, we exploit this characteristic to display information in the peripheral vision of the user. We introduce a mobile peripheral vision model, which can be used on any smart glasses with a head-mounted display without any additional hardware requirement. This model taps into the blocked peripheral vision of a user and simplifies multi-tasking when using smart glasses. To display the potential applications of this model, we implement an application for indoor and outdoor navigation. We conduct an experiment on 20 people on both smartphone and smart glass to evaluate our model on indoor and outdoor conditions. Users report to have spent at least 50% less time looking at the screen by exploiting their peripheral vision with smart glass. 90% of the users Agree that using the model for navigation is more practical than standard navigation applications.

In the smartphone era, instance recognition (IR) applications are widely used on mobile devices. However, IR applications are computationally expensive for a single mobile device, and usually they are delegated to back end servers. Nevertheless, even by exploiting the computational power offered by cloud services, IR tasks are not executed in real-time (i.e., in less than 30ms), which is crucial for interactive mobile applications. In this work, we present caching for instance recognition on cloud environments (CIRCE), a similarity caching (SC) framework designed to improve the performance of mobile IR applications in terms of execution time. Additionally, we introduce a parallel version of the Hessian-Affine detector combined with the SIFT descriptor, and a novel cache hit threshold (CHT) algorithm. Finally, we present two new public image datasets that we create to evaluate CIRCE. By using a cache size of just few hundreds, we obtain a hit ratio of at least 66% and a precision of at least 97%. In case of a cache hit, our system performs IR tasks in at most 14ms on three different applications.

Mobile augmented reality (MAR) applications are gaining popularity due to the wide adoption of mobile and especially wearable devices. Such devices often present limited hardware capabilities while MAR applications often rely on computationally intensive computer vision algorithms with extreme latency requirements. To compensate for the lack of computing power, offloading data processing to a distant machine is often desired. However, if this process introduces new constrains in the application, especially in terms of latency and bandwidth. If current network infrastructures are not ready for such traffic, we envision that future wireless networks such as 5G will rapidly be saturated by resource hungry MAR applications. Moreover, due to the high variance of wireless networks, MAR applications should not rely only on the evolution of infrastructures. In this article, we analyze MAR applications and justify their need for accessing external infrastructure. After a review of the existing network infrastructures and protocols, we define guidelines for future real-time and multimedia transport protocols, with a focus on MAR offloading.

The Internet of Things (IoT) is connecting people and smart devices on a scale that was once unimaginable. One major challenge for the IoT is to handle vast amount of sensing data generated from the smart devices that are resource-limited and subject to missing data due to link or node failures. By exploring cloud computing with the IoT, we present a cloud-based solution that takes into account the link quality and spatio-temporal correlation of data to minimize energy consumption by selecting sensors for sampling and relaying data. We propose a multiphase adaptive sensing algorithm with belief propagation (BP) protocol (ASBP), which can provide high data quality and reduce energy consumption by turning on only a small number of nodes in the network. We formulate the sensor selection problem and solve it using both constraint programming (CP) and greedy search. We then use our message passing algorithm (BP) for performing inference to reconstruct the missing sensing data. ASBP is evaluated based on the data collected from real sensors. The results show that while maintaining a satisfactory level of data quality and prediction accuracy, ASBP can provide load balancing among sensors successfully and preserves 80% more energy compared with the case where all sensor nodes are actively involved.

Task Graph (ATaG) is a sensor network application development paradigm where the application is visually described by a graph where the nodes correspond to application-level tasks and edges correspond to data flows. We extend ATaG with the option to add non-functional requirements: constraints on end-to-end delay and packet delivery rate. Setting up these constraints at the design phase naturally leads to enabling run-time assurance at the deployment phase, when the conditions of the constraints are used as network's performance goals. We provide both run-time middleware that checks the conditions of these constraints and a central management unit that dynamically adapts the system by doing task reallocation and putting task copies on redundant nodes. Through extensive simulations we show that the system is efficient enough to enable adaptations within tens of seconds even in large networks.

It is common to deploy stationary sensors in large geographical environments for monitoring purposes. In such cases, the monitored data are subject to data loss due to poor link quality or node failures. Fortunately, the sensing data are highly correlated both spatially and temporally. In this paper, we consider such networks in general, and jointly take into account the link quality estimates, and the spatio-temporal correlation of the data to minimise energy consumption by selecting sensors for sampling and relaying data. In particular, we propose a multi-phase adaptive sensing algorithm with belief propagation protocol (ASBP), which can provide high data quality and reduce energy consumption by turning on only a small number of nodes in the network. We explore the correlation of data, formulate the sensor selection problem and solve it using constraint programming (CP) and greedy search. Bayesian inference technique is used to reconstruct the missing sensing data. We show that while maintaining a satisfactory level of data quality and prediction accuracy, ASBP successfully provides load balancing among sensors and preserves 80% more energy compared to the case where all sensor nodes are actively involved.

Wireless sensor networks have become increasingly popular for environmental and activity monitoring, such as temperature, pollution, parking space, traffic, and crowd monitoring. Mobile users can collect and visualise sensing data by communicating with wireless sensors along their walks using Bluetooth or NFC. They can also share the sensing data on the Internet through 3G or WiFi connectivity. Nevertheless, mobile users may not be able to collect all the data from the sensors due to limited contact times and batteries. It is crucial to collect data with a maximum amount of information from the available resources. In this paper, we tackle the problem by prioritising the sensing data to maximise the data utility considering the quality of information of the sensing data and the communication overhead. We formulate the optimisation problem and propose a greedy algorithm for clustering the sensors and scheduling the data collection. Our greedy algorithm coordinates the mobile users in the sensing field in order to avoid the collection of redundant sensing data. We evaluate the data utility and energy consumption of the proposed algorithm using real mobility traces from the North Carolina state fair. The results demonstrate that our algorithm can significantly improve data utility at low communication overhead compared with an existing algorithm.

We consider a novel application in wireless sensor networks where mobile phones and wireless sensors can collaborate to collect sensing data. Although mobile phones can perform sensing at different locations, it is a challenge to provide stable sensing quality and availability over the entire area. One approach is to deploy stationary sensors at specific locations to maintain the sensing quality and availability. In this paper, we present a mathematical programming model to minimise the deployment cost by placing a minimum number of sensors at optimal locations. The problem is modelled by integer linear programming considering the sensing capabilities of both the mobile phones and wireless sensors. We evaluated the performance of our solution in terms of sensing quality, number of required sensors, and computation time. The results demonstrate that our approach satisfies the required sensing quality with optimal number of sensors in small sensing fields. It achieves near optimal solution with low computation time for large sensing fields.