Recent research on video moment retrieval has mostly focused on enhancing the performance of accuracy, efficiency, and robustness, all of which largely rely on the abundance of high-quality annotations. While the precise frame-level annotations are time-consuming and cost-expensive, few attentions have been paid to the labeling process. In this work, we explore a new interactive manner to stimulate the process of human-in-the-loop annotation in video moment retrieval task. The key challenge is to select “ambiguous” frames and videos for binary annotations to facilitate the network training. To be specific, we propose a new hierarchical uncertainty-based modeling that explicitly considers modeling the uncertainty of each frame within the entire video sequence corresponding to the query description, and selecting the frame with the highest uncertainty. Only selected frame will be annotated by the human experts, which can largely reduce the workload. After obtaining a small number of labels provided by the expert, we show that it is sufficient to learn a competitive video moment retrieval model in such a harsh environment. Moreover, we treat the uncertainty score of frames in a video as a whole, and estimate the difficulty of each video, which can further relieve the burden of video selection. In general, our active learning strategy for video moment retrieval works not only at the frame level but also at the sequence level. Experiments on two public datasets validate the effectiveness of our proposed method. Our code is released at https://github.com/renjie-liang/HUAL.

Given a descriptive language query, Video Moment Retrieval (VMR) aims to seek the corresponding semantic-consistent moment clip in the video, which is represented as a pair of the start and end timestamps. Although current methods have achieved satisfying performance, training these models heavily relies on the fully-annotated VMR datasets. Nonetheless, precise video temporal annotations are extremely labor-intensive and ambiguous due to the diverse preferences of different annotators.

Panoptic Scene Graph Generation (PSG) translates visual scenes to structured linguistic descriptions, i.e., mapping visual instances to subjects/objects, and their relationships to predicates. However, the annotators' preferences and semantic overlaps between predicates inevitably lead to the semantic mappings of multiple predicates to one relationship, i.e., biased-predicate annotations. As a result, with the contradictory mapping between visual and linguistics, PSG models are struggled to construct clear decision planes among predicates, so as to cause existing poor performances. Obviously, it is essential for the PSG task to tackle this multi-modal contradiction. Therefore, we propose a novel method that utilizes unbiased visual predicate representations for Biased-Annotation Identification (BAI) as a fundamental step for PSG/SGG tasks. Our BAI includes three main steps: predicate representation extraction, predicate representation debiasing, and biased-annotation identification. With flexible biased annotation processing methods, our BAI can act as a fundamental step of dataset debiasing. Experimental results demonstrate that our proposed BAI has achieved state-of-the-art performance, which promotes the performance of benchmark models to various degrees with ingenious biased annotation processing methods. Furthermore, our BAI shows great generalization and effectiveness on multiple datasets. Our codes are released at https://github.com/lili0415/BAI.

In this article, we investigate the energy efficiency (EE) maximization problem of the multihop intelligent reflecting surfaces (IRSs) assisted millimeter-wave (mmWave) Internet of Things (IoT) system. In this system, the base station (BS) provides services to IoT devices through the visible link established by the cascaded reflection of multihop IRSs. We aim to maximize the EE by jointly optimizing the active beamforming vector (ABV) at the BS, the number of active reflect elements (REs), the BS’s transmit power, and the phase shift (PS) at the multihop IRSs. To achieve this target, we formulate a nonlinear fractional programming problem. However, the formulated problem is nonconvex and all the variables are deeply coupled. To solve this problem, by using the sparse scattering property of the mmWave channels and the maximum ratio transmission method, we first transform the optimization of ABV at the BS into the optimization of the BS’s transmit power. We then propose an alternating optimization-based method to decouple all variables iteratively. Specifically, for the PS at multihop IRSs, we employ the sparse scattering property of the mmWave channels to divide it into multiple convex problems and obtain the optimal solution. For the number of active REs, we use the successive convex approximation method to obtain a suboptimal solution. For the BS’s transmit power, we employ the Dinkelbach method iteratively solving it and obtaining an optimal solution. Finally, we obtain an iterative suboptimal solution. The simulation results show that the EE of the proposed scheme outperforms the conventional schemes.