Current methods for disaster scene interpretation in remote sensing images (RSIs) mostly focus on isolated tasks such as segmentation, detection, or visual question-answering (VQA). However, these methods often fail to provide comprehensive and actionable insights, particularly in scenarios that demand the integration of multiple perception methods and specialized tools to address complex, multilayered challenges in geophysical disaster analysis. To fill this gap, this article introduces adaptive disaster interpretation (ADI), a novel task designed to solve requests by planning and executing multiple sequentially correlative interpretation tasks to provide a comprehensive analysis of disaster scenes. To facilitate research and application in this area, we present a new dataset named RescueADI, which contains high-resolution RSIs with annotations for three connected aspects: planning, perception, and recognition. The dataset includes 4044 RSIs, 16949 semantic masks, 14483 object bounding boxes, and 13424 interpretation requests across nine challenging request types. Moreover, we propose a new disaster interpretation method employing autonomous agents driven by large language models (LLMs) for task planning and execution, proving its efficacy in handling complex disaster interpretations. The proposed agent-based method solves various complex interpretation requests such as counting, area calculation, and path finding without human intervention, which traditional single-task approaches cannot handle effectively. Experimental results on RescueADI demonstrate the feasibility of the proposed task and show that our method achieves an accuracy 9% higher than existing VQA methods, highlighting its advantages over conventional disaster interpretation approaches.

Continual learning (CL) breaks off the one-way training manner and enables a model to adapt to new data, semantics and tasks continuously. However, current CL methods mainly focus on single tasks. Besides, CL models are plagued by catastrophic forgetting and semantic drift since the lack of old data, which often occurs in remote-sensing interpretation due to the intricate fine-grained semantics. In this paper, we propose Continual Panoptic Perception (CPP), a unified continual learning model that leverages multi-task joint learning covering pixel-level classification, instance-level segmentation and image-level perception for universal interpretation in remote sensing images. Concretely, we propose a collaborative cross-modal encoder (CCE) to extract the input image features, which supports pixel classification and caption generation synchronously. To inherit the knowledge from the old model without exemplar memory, we propose a task-interactive knowledge distillation (TKD) method, which leverages cross-modal optimization and task-asymmetric pseudo-labeling (TPL) to alleviate catastrophic forgetting. Furthermore, we also propose a joint optimization mechanism to achieve end-to-end multi-modal panoptic perception. Experimental results on the fine-grained panoptic perception dataset validate the effectiveness of the proposed model, and also prove that joint optimization can boost sub-task CL efficiency with over 13% relative improvement on panoptic quality. The project page is available at https://github.com/YBIO/CPP.

Current remote-sensing interpretation models often focus on a single task such as detection, segmentation, or caption. However, the task-specific designed models are unattainable to achieve the comprehensive multi-level interpretation of images. The field also lacks support for multi-task joint interpretation datasets. In this paper, we propose Panoptic Perception: a novel task and a new fine-grained dataset (FineGrip) to achieve a more thorough and universal interpretation for RSIs. The new task: 1) integrates pixel-level, instance-level, and image-level information for universal image perception, 2) captures image information from coarse to fine granularity, achieving deeper scene understanding and description, and 3) enables various independent tasks to complement and enhance each other through multi-task learning. By emphasizing multi-task interactions and the consistency of perception results, this task enables the simultaneous processing of fine-grained foreground instance segmentation, background semantic segmentation, and global fine-grained image captioning. Concretely, the FineGrip dataset includes 2,649 remote sensing images, 12054 fine-grained instance segmentation masks belonging to 20 foreground things categories, and 7599 background semantic masks for 5 stuff classes. Furthermore, we propose a joint optimization-based panoptic perception model. Experimental results on FineGrip demonstrate the feasibility of the panoptic perception task and the beneficial effect of multi-task joint optimization on individual tasks. The dataset will be publicly available.

As a cutting-edge challenge in the field of disaster evaluation, the detection of disasters in remote sensing images is crucial. However, most existing approaches to disaster detection simply solve the problem as a naive multi-class change detection, lacking accurate damage-level classification. In this paper, we propose a new approach to disaster detection called multi-level disaster detection (MLDD) that focuses on fine-grained damage-level classification. Our proposed approach tackles MLDD through hierarchical-correlation modeling and presents a universal disaster detection architecture. Specifically, we summarize two existing applicative methods, one-step training and pre-training, which are compatible with our proposed architecture. In addition, we propose two novel hierarchical approaches, namely the multi-task (MT) based and graph-encoding (GE) based approaches. The MT approach resolves MLDD through layer-wise learning in a progressive manner, building explicit multi-stage and implicit joint models to probe into the coarse-to-fine correlation for damage-level evaluation. The GE approach enhances hierarchical relationships by encoding multifold messaging directions and probabilities using a graph neural network. Furthermore, all four hierarchical paradigms can be embedded in our hierarchical MLDD architecture, which outperforms state-of-the-art methods on the xBD dataset, particularly in fine-grained damage-level classification. Overall, our proposed approach represents a significant improvement over existing disaster detection methods and has the potential to advance the field of disaster evaluation.