Quality analysis of weather forecasts is an essential topic in meteorology. Although traditional score-based evaluation metrics can quantify certain forecast errors, they are still far from meteorological experts in terms of descriptive capability, interpretability, and understanding of dynamic evolution. With the rapid development of Multi-modal Large Language Models (MLLMs), these models become potential tools to overcome the above challenges. In this work, we introduce an MLLM-based weather forecast analysis method, RadarQA, integrating key physical attributes with detailed assessment reports. We introduce a novel and comprehensive task paradigm for multi-modal quality analysis, encompassing both single frame and sequence, under both rating and assessment scenarios. To support training and benchmarking, we design a hybrid annotation pipeline that combines human expert labeling with automated heuristics. With such an annotation method, we construct RQA-70K, a large-scale dataset with varying difficulty levels for radar forecast quality evaluation. We further design a multi-stage training strategy that iteratively improves model performance at each stage. Extensive experiments show that RadarQA outperforms existing general MLLMs across all evaluation settings, highlighting its potential for advancing quality analysis in weather prediction.

Motion modeling is crucial in modern action recognition methods. As motion dynamics like moving tempos and action amplitude may vary a lot in different video clips, it poses great challenge on adaptively covering proper motion information. To address this issue, we introduce a Motion Diversification and Selection (MoDS) module to generate diversified spatio-temporal motion features and then select the suitable motion representation dynamically for categorizing the input video. To be specific, we first propose a spatio-temporal motion generation (StMG) module to construct a bank of diversified motion features with varying spatial neighborhood and time range. Then, a dynamic motion selection (DMS) module is leveraged to choose the most discriminative motion feature both spatially and temporally from the feature bank. As a result, our proposed method can make full use of the diversified spatio-temporal motion information, while maintaining computational efficiency at the inference stage. Extensive experiments on five widely-used benchmarks, demonstrate the effectiveness of the method and we achieve state-of-the-art performance on Something-Something V1 & V2 that are of large motion variation.

In this paper, we develop a large-scale visionlanguage fish benchmark, namely WildFish++, for comprehensive studies in multimedia research. Concretely, WildFish++ consists of 2,348 fish categories with 103,034 images in the wild, and 3,817 fish descriptions with 213,858 words. Based on these distinct characteristics, we mainly introduce four challenging research tasks on WildFish++. (1) Fine-Grained Recognition with Comparison Texts. WildFish++ naturally contains subtle difference among fish categories, which leads to fine-grained classification. Most approaches resort to tackle this problem by capturing discriminative regions in the view of each image. However, this paradigm may be still far way from extracting the most distinct features when the context on visual difference is not available. In this case, we propose to introduce comparison fish descriptions, a unique corpus that can directly point out subtle difference between highly-confused species and naturally serve as a kind of valuable context information. With such texts, we creatively elaborate a multi-modal fish network, aiming at incorporating those comparison textual information as prior knowledge and consequently leveraging it to guide CNNs to find subtle yet distinct regions in the context of comparison texts. (2) Open-Set Classification. We often confront with unknown categories in practice, e.g., there may still exist unknown fishes in our planet. Hence, we creatively adapt WildFish++ for a novel open-set classification task, which aims at correctly assigning each test image into the unknown class or one of known classes. More importantly, we investigate a number of practical designs to boost accuracy of deep learning models in open-set scenarios. (3) Cross-Modal Retrieval. WildFish++ not only contains diversified fish images in the wild but also has rich fish descriptions about morphology diagnosis, biology information, etc. Hence, we design a challenging cross-modal retrieval task, which leverages three subtasks such as text-to-text, text-to-image, image-to-text retrieval in a unified end-to-end framework. (4) Automatic Fish Classification. Automatic fish classification is a long-term research in marine biology, while current studies are unsatisfactory due to the lack of large-scale data. In this case, we train a number of CNNs with WildFish++, and use its pre-trained models to boost fish classification on most existing benchmarks of wild fishes. We will release WildFish++ with codes/protocols (https://github.com/PeiqinZhuang/WildFish++). We believe it can promote relevant studies in multimedia and beyond.

Fine-grained classification is a challenging problem, due to subtle differences among highly-confused categories. Most approaches address this difficulty by learning discriminative representation of individual input image. On the other hand, humans can effectively identify contrastive clues by comparing image pairs. Inspired by this fact, this paper proposes a simple but effective Attentive Pairwise Interaction Network (API-Net), which can progressively recognize a pair of fine-grained images by interaction. Specifically, API-Net first learns a mutual feature vector to capture semantic differences in the input pair. It then compares this mutual vector with individual vectors to generate gates for each input image. These distinct gate vectors inherit mutual context on semantic differences, which allow API-Net to attentively capture contrastive clues by pairwise interaction between two images. Additionally, we train API-Net in an end-to-end manner with a score ranking regularization, which can further generalize API-Net by taking feature priorities into account. We conduct extensive experiments on five popular benchmarks in fine-grained classification. API-Net outperforms the recent SOTA methods, i.e., CUB-200-2011 (90.0%), Aircraft (93.9%), Stanford Cars (95.3%), Stanford Dogs (90.3%), and NABirds (88.1%).

Fish recognition is an important task to understand the marine ecosystem and biodiversity. It is often challenging to identify fish species in the wild, due to the following difficulties. First, most fish benchmarks are small-scale, which may limit the representation power of machine learning models. Second, the number of fish species is huge, and there may still exist unknown categories in our planet. The traditional classifiers often fail to deal with this open-set scenario. Third, certain fish species are highly-confused. It is often hard to figure out the subtle differences, only by the unconstrained images. Motivated by these facts, we introduce a large-scale WildFish benchmark for fish recognition in the wild. Specifically, we make three contributions in this paper. First, WildFish is the largest image data set for wild fish recognition, to our best knowledge. It consists of 1000 fish categories with 54,459 unconstrained images, allowing to train high-capacity models for automatic fish classification. Second, we propose a novel open-set fish classification task for realistic scenarios, and investigate the open-set deep learning framework with a number of practical designs. Third, we propose a novel fine-grained recognition task, with the guidance of pairwise textual descriptions. Via leveraging the comparison knowledge in the sentence, we design a multi-modal fish net to effectively distinguish two confused categories in a pair. Finally, we release WildFish (https://github.com/PeiqinZhuang/WildFish), in order to bring benefit to more research studies in multimedia and beyond.