We introduce Lumos, the first end-to-end multimodal question-answering system with text understanding capabilities. At the core of Lumos is a Scene Text Recognition (STR) component that extracts text from first person point-of-view images, the output of which is used to augment input to a Multimodal Large Language Model (MM-LLM). While building Lumos, we encountered numerous challenges related to STR quality, overall latency, and model inference. In this paper, we delve into those challenges, and discuss the system architecture, design choices, and modeling techniques employed to overcome these obstacles. We also provide a comprehensive evaluation for each component, showcasing high quality and efficiency.

Video based 3D reconstruction of human motion plays an important role in many applications. We implement a system that robustly reconstructs 3D human motion from markerless videos taken by a single camera. Our system only requires a desktop PC and a mainstream camera, and doesn't involve complex camera calibration, making it easy to implement and widely accessible in daily uses such as human computer interaction or entertainment.