3D spatial reasoning in dynamic, audio-visual environments is a cornerstone of human cognition yet remains largely unexplored by existing Audio-Visual Large Language Models (AV-LLMs) and benchmarks, which predominantly focus on static or 2D scenes. We introduce SAVVY-Bench, the first benchmark for 3D spatial reasoning in dynamic scenes with synchronized spatial audio. SAVVY-Bench is comprised of thousands of carefully curated question–answer pairs probing both directional and distance relationships involving static and moving objects, and requires fine-grained temporal grounding, consistent 3D localization, and multi-modal annotation. To tackle this challenge, we propose SAVVY, a novel training-free reasoning pipeline that consists of two stages: (i) Egocentric Spatial Tracks Estimation, which leverages AV-LLMs as well as other audio-visual methods to track the trajectories of key objects related to the query using both visual and spatial audio cues, and (ii) Dynamic Global Map Construction, which aggregates multi-modal queried object trajectories and converts them into a unified global dynamic map. Using the constructed map, a final QA answer is obtained through a coordinate transformation that aligns the global map with the queried viewpoint. Empirical evaluation demonstrates that SAVVY substantially enhances performance of state-of-the-art AV-LLMs, setting a new standard and stage for approaching dynamic 3D spatial reasoning in AV-LLMs.

Hybrid Optical Neural Networks (ONNs, typically consisting of an optical frontend and a digital backend) offer an energy-efficient alternative to fully digital deep networks for real-time, power-constrained systems. However, their adoption is limited by two main challenges: the accuracy gap compared to large-scale networks during training, and discrepancies between simulated and fabricated systems that further degrade accuracy. While previous work has proposed end-to-end optimizations for specific datasets (e.g., MNIST) and optical systems, these approaches typically lack generalization across tasks and hardware designs. To address these limitations, we propose a task-agnostic and hardware-agnostic pipeline that supports image classification and segmentation across diverse optical systems. To assist optical system design before training, we design the metasurface layout based on fabrication constraints. For training, we introduce Neural Tangent Knowledge Distillation (NTKD), which aligns optical models with electronic teacher networks, thereby narrowing the accuracy gap. After fabrication, NTKD also guides fine-tuning of the digital backend to compensate for implementation errors. Experiments on multiple datasets (e.g., MNIST, CIFAR, Carvana Masking) and hardware configurations show that our pipeline consistently improves ONN performance and enables practical deployment in both pre-fabrication simulations and physical implementations.

Endoscopic assessment using the Mayo clinic score (or Mayo score, 4 categories) is currently the standard for diagnosing and evaluating mucosal disease activities. However, annotating Mayo scores is time-consuming and often relies on weakly labeled evaluations from central and local readers (doctors), leading to a large number of unlabeled or mislabeled video clips. Additionally, such labels also suffer from data imbalance due to patient distributions and varying disease severity levels. This gap underscores the need for more customizable and refined methods for endoscopic scoring and localization. To address these challenges, we introduce an end-to-end pipeline with a new dataset for endoscopic scoring and localization in unconstrained clinical trial videos. Specifically, we propose an automated scoring system that includes an active learning-based preprocessing stage for cleaning the raw videos, which are then weakly labeled by doctors. Therefore, we obtain a comprehensive dataset for endoscopic Mayo scoring, comprising approximately 49.9 hours of video and 86, 423 clips, providing a solid foundation for developing endoscopic scoring and localization models. Then, we propose a video classification model for Mayo score classifications. For personalized disease quantification in localization, we introduce a novel 1D trajectory model with a novel cumulative disease score that addresses the limitations of previous 3D trajectory projection methods. Our dataset and end-to-end pipeline offer a valuable foundation for advancing endoscopic clinical trial research.