Recent advances in 3D human motion and language integration have primarily focused on text-to-motion generation, leaving the task of motion understanding relatively unexplored. We introduce Dense Motion Captioning, a novel task that aims to temporally localize and caption actions within 3D human motion sequences. Current datasets fall short in providing detailed temporal annotations and predominantly consist of short sequences featuring few actions. To overcome these limitations, we present the Complex Motion Dataset (CompMo), the first large-scale dataset featuring richly annotated, complex motion sequences with precise temporal boundaries. Built through a carefully designed data generation pipeline, CompMo includes 60,000 motion sequences, each composed of multiple actions ranging from at least two to ten, accurately annotated with their temporal extents. We further present DEMO, a model that integrates a large language model with a simple motion adapter, trained to generate dense, temporally grounded captions. Our experiments show that DEMO substantially outperforms existing methods on CompMo as well as on adapted benchmarks, establishing a robust baseline for future research in 3D motion understanding and captioning.

Camera relocalization is a crucial problem in computer vision and robotics. Recent advancements in neural radiance fields (NeRFs) have shown promise in synthesizing photo-realistic images. Several works have utilized NeRFs for refining camera poses, but they do not account for lighting changes that can affect scene appearance and shadow regions, causing a degraded pose optimization process. In this paper, we propose a two-staged pipeline that normalizes images with varying lighting and shadow conditions to improve camera relocalization. We implement our scene representation upon a hash-encoded NeRF which significantly boosts up the pose optimization process. To account for the noisy image gradient computing problem in grid-based NeRFs, we further propose a re-devised truncated dynamic low-pass filter (TDLF) and a numerical gradient averaging technique to smoothen the process. Experimental results on several datasets with varying lighting conditions demonstrate that our method achieves state-of-the-art results in camera relocalization under varying lighting conditions. Code and data will be made publicly available.

Deep learning (DL) technology has made breakthroughs in a wide range of intelligent tasks, such as vision, language, recommendation systems, and so on. Sparse matrix multiplication (SpMM) is the key computation kernel of most sparse models. Conventional computing platforms, such as CPUs, GPUs, and AI chips with regular processing units, are unable to effectively support sparse computation due to their fixed structure and instruction sets. This work extends Sparkle, an accelerator architecture, which is developed specifically for processing SpMM in DL. During the balanced data loading process, some modifications are implemented to enhance the flexibility of the Sparkle architecture. Additionally, a Sparkle generator is proposed to accommodate diverse resource constraints and facilitate adaptable deployment. Leveraging Sparkle’s structural parameters and template-based design methods, the generator enables automatic Sparkle circuit generation under varying parameters. An instantiated Sparkle accelerator is implemented on the Xilinx xqvu11p FPGA platform with a specific configuration. Compared to the state-of-the-art SpMM accelerator SIGMA, the Sparkle accelerator instance improves the sparse computing efficiency by about 10 to 20 \(\%\) . Furthermore, the Sparkle instance achieved 7.76 \(\times\) higher performance over the Nvidia Orin NX GPU. More instances of accelerators with different parameters were evaluated, demonstrating that the Sparkle architecture can effectively accelerate SpMM.

Deep learning (DL) technology is applied to a wide range of intelligent tasks across vision, language, recommendation systems, etc. Large DL models with high sparsity become critical for various intelligent applications and require an energy-efficient hardware accelerator. Sparse-dense matrix multiplication (SpMM) is a key computation kernel widely used in most sparse and large DL workloads. However, traditional computing platforms such as CPU, GPU, and Al chips with regular processing units are limited to support sparsity by their fixed structures. In this work, a specific SpMM accelerator named Sparkle is proposed which achieves high performance and high computational efficiency. A block-wise arrangement approach is proposed in Sparkle to process matrix multiplications. A novel compressed sparsity format, the pointer-bitmap, is designed to simplify the decoding process and improve the efficiency of data loading. Grouped PEs and configurable hierarchical reduction network are deployed to leverage sparsity, further enhancing the utilization of the compute resources. Sparkle is implemented using the Xilinx xqvu11p FPGA. A diverse set of matrices in DL workloads are evaluated and Sparkle achieves 2.1× higher energy efficiency over the NVIDIA TITAN X GPU. Our experiments also show that Sparkle roughly promotes 26% compute efficiency better than state-of-the-art sparse accelerators SIGMA.