Abstract Dense activity detection is a subtask of activity detection that aims to localise and identify multiple human activities in video clips. Existing methods adopt offline frameworks that require video frames to be available when activity detection begins. These offline methods are unable to be applied to online scenarios. An online framework is proposed for dense activity detection. The framework has two stages: warm‐up and detection. Warm‐up is the initialisation of dense activity detection, which generates a contextual model called an online aggregated‐event. After that, the method moves into the detection stage, which consists of two modules: coarse label prediction and refined label prediction. Coarse label prediction predicts activity labels by taking the online aggregated‐event as a priori; then, prediction is refined by two techniques, human–object interaction detection and online relation reasoning. The proposed method is evaluated using two dense activity datasets: Charades and AVA. The experimental results show that the proposed method has better performance than existing offline methods after the whole video input is added to the algorithm.

With the development of deep learning, the design of an appropriate network structure becomes fundamental. In recent years, the successful practice of Neural Architecture Search (NAS) has indicated that an automated design of the network structure can efficiently replace the design performed by human experts. Most NAS algorithms make the assumption that the overall structure of the network is linear and focus solely on accuracy to assess the performance of candidate networks. This paper introduces a novel NAS algorithm based on a multi-objective modeling of the network design problem to design accurate Convolutional Neural Networks (CNNs) with a small structure. The proposed algorithm makes use of a graph-based representation of the solutions which enables a high flexibility in the automatic design. Furthermore, the proposed algorithm includes novel ad-hoc crossover and mutation operators. We also propose a mechanism to accelerate the evaluation of the candidate solutions. Experimental results demonstrate that the proposed NAS approach can design accurate neural networks with limited size.

The task of referring expression comprehension (REC) is to localise an image region of a specific object described by a natural language expression, and all existing REC methods assume that the object described by the referring expression must be located in the given image. However, this assumption is not correct in some real applications. For example, a visually impaired user might tell his robot ‘please take the laptop on the table to me’. In fact, the laptop is not on the table anymore. To address this problem, the authors propose a novel REC model to deal with the situation where expression‐image mismatching occurs and explain the mismatching by linguistic feedback. The authors' REC model consists of four modules: the expression parsing module, the entity detection module, the relationship detection module, and the matching detection module. They built a data set called NP‐RefCOCO+ from RefCOCO+ including both positive samples and negative samples. The positive samples are original expression‐image pairs in RefCOCO+. The negative samples are the expression‐image pairs in RefCOCO+, whose expressions are replaced. They evaluate the model on NP‐RefCOCO+ and the experimental results show the advantages of their method for dealing with the problem of expression‐image mismatching.