Image captioning has long been a pivotal task in visual understanding, with recent advancements in vision-language models (VLMs) significantly enhancing the ability to generate detailed image captions. However, the evaluation of detailed image captioning remains underexplored due to outdated evaluation metrics and coarse annotations. In this paper, we introduce DeCapBench along with a novel metric, DCScore, specifically designed for detailed captioning tasks. DCScore evaluates hallucinations and fine-grained comprehensiveness by deconstructing responses into the smallest self-sufficient units, termed primitive information units, and assessing them individually. Our evaluation shows that DCScore aligns more closely with human judgment than other rule-based or model-based metrics. Concurrently, DeCapBench exhibits a high correlation with VLM arena results on descriptive tasks, surpassing existing benchmarks for vision-language models. Additionally, we present an automatic fine-grained feedback collection method, FeedQuill, for preference optimization based on our advanced metric, demonstrating robust generalization capabilities across auto-generated preference data. Extensive experiments on multiple VLMs demonstrate that our method not only significantly reduces hallucinations but also enhances performance across various benchmarks, achieving superior detail captioning performance while surpassing GPT-4o.

Building large-scale generalist pre-trained models for many tasks is becoming an emerging and potential direction in reinforcement learning (RL).Research such as Gato and Multi-Game Decision Transformer have displayed outstanding performance and generalization capabilities on many games and domains.However, there exists a research blank about developing highly capable and generalist models in multi-agent RL (MARL), which can substantially accelerate progress toward general AI.To fill this gap, we propose Multi-Agent multi-Game ENtity TrAnsformer (MA-GENTA) from the entity perspective as orthogonal research to previous time-sequential modeling.Specifically, to deal with different state/observation spaces in different games, we analogize games as languages by aligning one single game to one single language, thus training different "tokenizers" and a shared transformer for various games.The feature inputs are split according to different entities and tokenized in the same continuous space.Then, two types of transformer-based models are proposed as permutationinvariant architectures to deal with various numbers of entities and capture the attention of different entities.MAGENTA is trained on

The existing image embedding networks are basically vulnerable to malicious attacks such as JPEG compression and noise adding, not applicable for real-world copyright protection tasks. To solve this problem, we introduce a generative deep network based method for hiding images into images while assuring high-quality extraction from the destructive synthesized images. An embedding network is sequentially concatenated with an attack layer, a decoupling network and an image extraction network. The addition of decoupling network learns to extract the embedded secret image from the attacked image. We also pinpoint the weaknesses of the adversarial training for robustness in previous works and build our improved real-world attack simulator. Experimental results demonstrate the superiority of the proposed method against typical digital attacks by a large margin, as well as the performance boost of the recovered images with the aid of progressive recovery strategy. Besides, we are the first to robustly hide three secret images.

The security of data storage is a big issue for companies. They must take effective steps to prevent valuable image datasets from being stolen for illegal commercial purposes. While data encryption is a common solution, it drastically down-grades the visual quality and therefore forbids common yet trivial use such as eye-checking without a decryption. We present a novel solution for dataset protection in this scenario by robustly and reversibly transform the images into adver-sarial images. An invertible Image Dataset Protection NET-work (IDP-Net) is developed to introduce slight and acceptable changes to the images within the dataset. The protected images can be published and circulated on the social networks instead of their original version. Malicious attackers can only observe the images but cannot train pirated models based on them. Meanwhile, IDP-Net ensures the performance of au-thorized models, namely, trusted users can revert the protection and retrieve the protected images to their original version. Therefore, the dataset can be stored within the pro-tected version alone to ensure safety. Extensive experiments demonstrate that IDP-Net can better protect the security of image dataset against defensive methods compared to previ-ous methods. Besides, the introduced distortion is acceptable and the original images can be reconstructed nearly error-free.