Mixed Reality (MR) telecollaboration aims to enable users to share local objects as real-time synchronized virtual replicas to remote partners and collaborate on physical tasks as if they were co-located. However, in everyday scenarios with mobile and easy-to-setup MR devices, visualizing shared objects in a single modality, ranging from 2D images to 3D reconstruction, struggles to simultaneously optimize all the aspects of Spatiality, Fidelity, and Real-time performance. To overcome this issue, existing methods explore integrating multiple visualization modalities to leverage their respective advantages in subsets of the three aspects. However, they focus on fixed modality combinations without considering user-centered task contexts and workflow, where users may prioritize different aspects of the visualization across task phases. Moreover, they lack support for switching or require manual switching across modalities, which could become disruptive and tiring. In this paper, we propose adapting object visualization based on spatiotemporal contexts in telecollaboration. Specifically, we first couple task type with the user's relative viewing distance as the spatial context, and examine its impact on users' prioritized visualization aspects, and the corresponding switching thresholds. With differing generation speeds of modalities, we then explore temporal switching schemes when the preferred modality is not immediately available. With the obtained design choices, we implement CollabVisAdapt, a proof-of-concept prototype that supports automatic adaptation of object visualization based on spatiotemporal contexts in MR telecollaboration. A user study in remote maintenance verifies the effectiveness of the proposed workflow with adaptive visualization and the usability of the system.

Augmented Reality (AR) teleconferencing allows spatially distributed users to interact with each other in 3D through agents in their own physical environments. Existing methods leveraging volumetric capturing and reconstruction can provide a high-fidelity experience but are often too complex and expensive for everyday use. Other solutions target mobile and effortless-to-setup teleconferencing on AR Head Mounted Displays (HMD). They directly transplant the conventional video conferencing onto an AR-HMD platform or use avatars to represent remote participants. However, they can only support either a high fidelity or a high level of co-presence. Moreover, the limited Field of View (FoV) of HMDs could further degrade users' immersive experience. To achieve a balance between fidelity and co-presence, we explore using life-size 2D video-based avatars (video avatars for short) in AR teleconferencing. Specifically, with the potential effect of FoV on users' perception of proximity, we first conducted a pilot study to explore the local-user-centered optimal placement of video avatars in small-group AR conversations. With the placement results, we then implement a proof-of-concept prototype of video-avatar-based teleconferencing. We conduct user evaluations with our prototype to verify its effectiveness in balancing fidelity and co-presence. Following the indication in the pilot study, we further quantitatively explore the effect of FoV size on the video avatar's optimal placement through a user study involving more FoV conditions in a VR-simulated environment. We regress placement models to serve as references for computationally determining video avatar placements in such teleconferencing applications on various existing AR HMDs and future ones with bigger FoVs.

Augmented Reality (AR) teleconferencing aims to enable remotely separated users to meet with each other in their own physical spaces as if they are face-to-face. Among all solutions, the video-avatar-based approach has the advantage of balancing fidelity and the sense of co-presence using easy-to-setup devices, including only a camera and an AR Head-Mounted Display (HMD). However, non-verbal cues indicating “who is looking at whom” are always lost or misdelivered in multiparty teleconferencing experiences. To make users aware of such non-verbal cues, existing solutions explore screen-based visualizations, incorporate additional hardware, or alter to use a virtual avatar representation. However, they lack immersion, are less feasible for everyday usage due to complex installations, or lose the fidelity of remote users’ authentic appearances. In this paper, we decompose such attention awareness into the awareness of being looked at and the awareness of attention between other users and address them in a decoupled process. Specifically, through a user study, we first find an unobtrusive and reasonable layout “Attention Circle” to retarget a looker’s head gaze to the one being looked at. We then conduct the second user study to find an effective and intuitive “rotatable 2.5D video avatar with attention thumbnail” visualization to aid users in being aware of other users’ attention. With the design choice distilled from the studies, we implement A3RT, a proof-of-concept prototype system that empowers attention-aware 2.5D-video-avatar-based multiparty AR teleconferencing in an easy, everyday setup. Ablation and usability studies on the prototype verify the effectiveness of our proposed components and the full system.

Avatar-mediated symmetric Augmented Reality (AR) telepresence has emerged with the ability to empower users located in different remote spaces to interact with each other in 3D through avatars. However, different spaces have heterogeneous structures and features, which bring difficulties in synchronizing avatar motions with real user motions and adapting avatar motions to local scenes. To overcome these issues, existing methods generate mutual movable spaces or retarget the placement of avatars. However, these methods limit the telepresence experience in a small sub-area space, fix the positions of users and avatars, or adjust the beginning/ending positions of avatars without presenting smooth transitions. Moreover, the delay between the avatar retargeting and users' real transitions can break the semantic synchronization between users' verbal conversation and perceived avatar motion. In this paper, we first examine the impact of the aforementioned transition delay and explore the preferred transition style with the existence of such delay through user studies. With the results showing a significant negative effect of avatar transition delay and providing the design choice of the transition style, we propose a Predict-and-Drive controller to diminish the delay and present the smooth transition of the telepresence avatar. We also introduce a grouping component as an upgrade to immediately calculate a coarse virtual target once the user initiates a transition, which could further eliminate the avatar transition delay. Once having the coarse virtual target or an exactly predicted target, we find the corresponding target for the avatar according to the pre-constructed mapping of objects of interest between two spaces. The avatar control component maintains an artificial potential field of the space and drives the avatar towards the target while respecting the obstacles in the physical environment. We further conduct ablation studies to evaluate the effectiveness of our proposed components.

A plethora of mobile video applications involving all aspects of social life are becoming increasingly prevalent. However, for resource-hungry and delay-sensitive multi-view video applications such as multi-channel video conferencing and 3D videos, the hardware resources of mobile terminals becomes a bottleneck in concurrently decoding multiple videos. Fusing multiple views into a single-view video stream in the cloud before transmission can unload the computation of mobile terminals. But the delay increment caused by video fusing in this cloudbased multi-view video streaming makes the strategy of deploying such applications a new area to examine. In this paper, in the interest of deployment with higher resource utilization and less latency, first, together with a load model of the fused sharable video interaction process, a channel admission control algorithm that targets maximizing the supportive capacity of the cloud is introduced. Subsequently, a channel deployment algorithm based on load coordination between the cloud and mobile clients is proposed to ensure that the delay caused by cloud processing is acceptable while minimizing the terminal computing load.

To further enhance the immersion perception of remote interaction, avatars can be involved harnessing Head Mounted Display (HMD) based Augmented Reality (AR). In our demonstration, we present an avatar based remote interaction system AvatarMeeting, enabling users to meet with remote peers through interactive personalized avatars just like face to face. Specifically, we propose a novel framework including a consumer-grade set-up, a complete transmission scheme and a processing pipeline, which consists of prescan modeling, pose detection and action reconstruction. And an angle based reconstruction approach is introduced to empower the AR avatars to perform the same actions as each remote real person do in real time smoothly while keeping a good avatar shape.

HTTP adaptive streaming (HAS) is the state-of-the-art technology for improving the quality of user experience under conditions of time-varying available bandwidth. Developing the bitrate adaptation algorithm becomes more challenging with the transition to 4G cellular networks. The features of bandwidth variation caused by changes in radio channel quality are significantly different from the pattern caused by changes in radio channel resources. In this paper, we propose a bitrate adaptation algorithm for 4G cellular networks with bandwidth variation pattern differentiation. By investigating the bandwidth data profiles collected from an actual 4G cellular network with the field test approach, we distinguish the pattern of bandwidth capacity variations as sustained fluctuations and instantaneous hopping. With bandwidth variation pattern differentiation, the proposed algorithm performs a smoothed bitrate adaptation to sustained fluctuations and an instant bitrate adaptation to instantaneous hopping. Performance evaluations obtained on a 4G cellular network testbed demonstrate that the algorithm achieves reductions in bitrate switching frequency and playback stalls while increasing the average bitrate.