Visual query localization (VQL) is pivotal for constructing episodic memory from egocentric videos. However, current methods often rely on computationally intensive attention mechanisms and rigid one-shot regression, which inherently struggle to model uncertainty and exhibit limited adaptability to degraded query appearances. This contrasts sharply with the human brain’s selective encoding and iterative hypothesis verification processes for episodic memory. Inspired by the human brain’s ability to selectively filter irrelevant information and reconstruct vague memory fragments through generative inference, we propose BiOVQL, a brain-inspired one-stage VQL framework. First, inspired by the hippocampus’s selective retention mechanism, we propose the Hippocampus-like Query-guided Spatio-Temporal Compression (HQSTC) module. By leveraging a selective state space model, visual queries are treated as neuromodulators, dynamically gating the video stream to maintain a compact latent state. This guides the model to consistently focus on query-relevant visual cues, enabling query-conditioned feature compression and efficient spatio-temporal memory encoding. Second, inspired by the prefrontal cortex’s re-localization mechanisms, we propose the Prefrontal-like Generative Refinement Localization (PGRL) module. We leverage a diffusion model to reconstruct the localization process as iterative denoising from noise to certainty, which aligns well with the human visual system’s coarse-to-fine perceptual reasoning. This enhances the model’s robustness in handling spatial ambiguities and achieving precise spatio-temporal retrieval. We conducted extensive experiments on the Ego4D-VQ benchmark, demonstrating that BiOVQL achieves state-of-the-art performance with comparable computational efficiency, thus offering an efficient and brain-inspired paradigm for VQL.

Video moment retrieval locates a specified moment by a sentence query. Recent approaches have made remarkable advancements with large-scale video-sentence annotations. These annotations require extensive human labor and expertise, leading to the need for unsupervised fashion. Generating pseudo-supervision from videos is an effective strategy. With the power of the large-scale pre-trained model, we introduce knowledge into constructing pseudo-supervision. The main technical challenge is improving pseudo-supervision diversity and alleviating noise brought by external knowledge. To address these problems, we propose two Knowledge-Based Pseudo-Supervision Construction (KPSC) strategies: KPSC-P and KPSC-F. They all follow two steps: generating diverse samples and alleviating knowledge chaos. The main difference is that the former first learns a representation space with prompt tuning, while the latter directly utilizes data information. KPSC-P has two modules: (1) Proposal Prompt (PP): Generate temporal proposals; (2) Verb Prompt (VP): Generate pseudo-queries with noun-verb patterns. KPSC-F also has two modules: (1) Captioner: Generating candidate queries; (2) Filter: Alleviating knowledge chaos. Thus, our KPSC involves two attempts to extract knowledge from pre-trained models. Extensive experiments show that our attempts outperform the existing unsupervised methods on two public datasets (Charades-STA and ActivityNet-Captions) and perform on par with several methods using stronger supervision.

Video moment retrieval (VMR) is a cutting-edge vision-language task locating a segment in a video according to the query. Though the methods have achieved significant performance, they assume that training and testing samples share the same action types, hindering real-world application. In this paper, we specifically consider a new problem: video moment retrieval by queries with unseen actions. We propose a plug-and-play structure, Routing Evidence (RE), with multiple evidence-learning heads and dynamically route one to locate a sentence with an unseen action. Each evidence-learning head estimates the uncertainty while regressing timestamps. We formulate the evidence distribution by a Normal-Inverse Gamma function and design a router to select the most appropriate distribution for a sample. Empirically, we study the efficacy of RE on three updated databases where training and testing samples contain different action types. We find that RE outperforms other state-of-the-art methods with a more robust predictor. Code and data will be available at https://github.com/dieuroi/Routing-Evidence.

Selecting elements for graphic design is essential for ensuring a correct understanding of clients' requirements as well as improving the efficiency of designers before a fine-designed process. Some semi-automatic design tools proposed layout templates where designers always select elements according to the rectangular boxes that specify how elements are placed. In practice, layout and element selection are complementary. Compared to the layout which can be readily obtained from pre-designed templates, it is generally time-consuming to mindfully pick out suitable elements, which calls for an automation of elements selection. To address this, we formulate element selection as a sequential decision-making process and develop a deep element selection network (DESN). Given a layout file with annotated elements, new graphical elements are selected to form graphic designs based on aesthetics and consistency criteria. To train our DESN, we propose an end-to-end, reinforcement learning based framework, where we design a novel reward function that jointly accounts for visual aesthetics and consistency. Based on this, visually readable and aesthetic drafts can be efficiently generated. We further contribute a layout-poster dataset with exhaustively labeled attributes of poster key elements. Qualitative and quantitative results indicate the efficacy of our approach.