Augmented reality (AR) offers users immersive experiences to interact with digital contents in their physical space. However, practical AR applications are challenged by the tight coupling of algorithm and engineering during the development and deployment phases as well as the execution requirements of hybrid AR subtasks on heterogeneous and resource-constraint mobile devices. In this work, we build an end-to-end, cross-platform, and efficient AR system, called ARSys. The infrastructure in ARSys adopts the new principle of integrated design, unifies and refines AR fundamental capabilities, supports streaming media processing, model inference, and real-time rendering by exposing high-performance tensor compute engine to top, and constructs a Python multi-instance virtual machine as the cross-platform AR task execution container. The runtime mechanism of ARSys schedules AR tasks in a pipeline parallelism way and allocates subtasks to hardware backends by optimizing the slowest node. The development workbench and the deployment platform in ARSys allow the decoupling of algorithms written in Python from engineering components in C/C++ and further support remote debugging and quick validation of AR algorithms. We extensively evaluate ARSys in practical AR applications across high-end, mid-end, and low-end Android and iOS devices, demonstrating higher development, deployment, and runtime efficiency than existing MediaPipe-oriented framework. ARSys has been integrated into Mobile Taobao for production use.

To provide full-body 3D videos of performers showcasing diverse clothes with dynamic movements for e-commerce platforms, we develop an end-to-end, low-cost, and high-fidelity production and deployment pipeline. We first set up a low-cost capture studio with only 24 RGB cameras and embrace fast neural surface reconstruction to produce high-quality meshes without depth information. We then quickly group all the frames with local motion priors, select a keyframe for each group, and accurately register any other frame to the keyframe under the guidance of semantic labels, thereby avoiding transmitting all the frames to mobile devices and loading them into memory. For real-time rendering, we propose an on-device sparse computation method for efficient deformation from keyframes to the other frames. Evaluation over 2 self-captured performances and 8 public performances reveals that the pipeline achieves the reconstruction time of 28 minutes per frame, the average PSNR of 30.4, the average bandwidth requirement of 4.2MB/s, and the on-device frame rate of 60 fps, demonstrating superiority over existing baselines.