Audio-visual generalized zero-shot learning (AV-GZSL) for video classification is a task where the model learns to identify unseen video classes from multimodal audio-visual inputs. This is a combination of two equally challenging tasks where one tries to perform video classification from two different modalities of inputs, and the other pushes to achieve this in a zero-shot setting. The natural alignment between audio and visual modalities is the key to addressing this relatively unexplored task. The predominant approach in AV-GZSL has been to learn better cross-modal attention between the two input domains and leverage large language pretraining. However, for better attention and pretraining, there exists a semantic gap between the embedding of different modalities that requires a more diverse and less sparse representation of the joint embedding space. To overcome this, we propose a complementary approach to the existing research direction, where we simulate unseen audio-visual features using a generative model, and regulate it by combining contrastive and discriminative loss. To demonstrate the effectiveness of our approach, we benchmark our model on VGGSound-GZSL, ActivityNet-GZSL, and UCF-GZSL and report state-of-the-art performance, and qualitatively show that unseen classes are better clustered together with our generative approach.

Dynamic neural networks can greatly reduce computation redundancy without compromising accuracy by adapting their structures based on the input. In this paper, we explore the robustness of dynamic neural networks against energy-oriented attacks targeted at reducing their efficiency. Specifically, we attack dynamic models with our novel algorithm GradMDM. GradMDM is a technique that adjusts the direction and the magnitude of the gradients to effectively find a small perturbation for each input, that will activate more computational units of dynamic models during inference. We evaluate GradMDM on multiple datasets and dynamic models, where it outperforms previous energy-oriented attack techniques, significantly increasing computation complexity while reducing the perceptibility of the perturbations https://github.com/lingengfoo/GradMDM.

Understanding animals' behaviors is significant for a wide range of applications. However, existing animal behavior datasets have limitations in multiple aspects, including limited numbers of animal classes, data samples and provided tasks, and also limited variations in environmental conditions and viewpoints. To address these limitations, we create a large and diverse dataset, Animal Kingdom, that provides multiple annotated tasks to enable a more thorough understanding of natural animal behaviors. The wild animal footages used in our dataset record different times of the day in extensive range of environments containing variations in backgrounds, viewpoints, illumination and weather conditions. More specifically, our dataset contains 50 hours of annotated videos to localize relevant animal behavior segments in long videos for the video grounding task, 30K video sequences for the fine-grained multi-label action recognition task, and 33K frames for the pose estimation task, which correspond to a diverse range of animals with 850 species across 6 major animal classes. Such a challenging and comprehensive dataset shall be able to facilitate the community to develop, adapt, and evaluate various types of advanced methods for animal behavior analysis. Moreover, we propose a Collaborative Action Recognition (CARe) model that learns general and specific features for action recognition with unseen new animals. This method achieves promising performance in our experiments. Our dataset can be found at https://sutdcv.github.io/Animal-Kingdom.