Multimodal visual language models are gaining prominence in open-world applications, driven by advancements in model architectures, training techniques, and high-quality data. However, their performance is often limited by insufficient task-specific data, leading to poor generalization and biased outputs. Existing efforts to increase task diversity in fine-tuning datasets are hindered by the labor-intensive process of manual task labeling, which typically produces only a few hundred task types. To address this, we propose TaskGalaxy, a large-scale multimodal instruction fine-tuning dataset comprising 19,227 hierarchical task types and 413,648 samples. TaskGalaxy utilizes GPT-4o to enrich task diversity by expanding from a small set of manually defined tasks, with CLIP and GPT-4o filtering those that best match open-source images, and generating relevant question-answer pairs. Multiple models are employed to ensure sample quality. This automated process enhances both task diversity and data quality, reducing manual intervention. Incorporating TaskGalaxy into LLaVA-v1.5 and InternVL-Chat-v1.0 models shows substantial performance improvements across 16 benchmarks, demonstrating the critical importance of task diversity. TaskGalaxy is publicly released at https://github.com/Kwai-YuanQi/TaskGalaxy.

Inspired by the recent progress in object detection (i.e., DETR), the set prediction mechanism significantly advances the research of semantic segmentation and achieves state-of-the-art performance on popular segmentation benchmarks. The generic pipeline of such a mechanism often firstly takes learnable query features to predict classes and segment masks separately and then blends these class-aware segment masks into the final segmentation mask. One key factor behind the successful training of this pipeline is to apply thebipartite matchingstrategy between the set of predictions and ground-truth segments. However, we find that thebipartite matching-based assignment often tends to segment one target class with only a few learnable queries, making many other pre-defined queries useless. In this article, we propose a simple way, namedDropQueries(DQ), to facilitate the set prediction based segmentation architectures. At each iteration of training, our DQ randomly and independently drops each learnable query with a certain probability beforebipartite matching. In this way, more queries are encouraged to participate in the segmentation process to discover comprehensive segment representations. We conduct extensive experiments using MaskFormer and Mask2Former as two basic yet powerful segmentation architectures. Without bells and whistles, our DQ strategy can bring consistent improvements over strong baselines on popular semantic segmentation benchmarks, including ADE 20 K, Cityscapes, COCO Stuff 10 K and VSPW.