Interleaved text-and-image generation has been an intriguing research direction, where the models are required to generate both images and text pieces in an arbitrary order.Despite the emerging advancements in interleaved generation, the progress in its evaluation still significantly lags behind.Existing evaluation benchmarks do not support arbitrarily interleaved images and text for both inputs and outputs, and they only cover a limited number of domains and use cases.Also, current works predominantly use similarity-based metrics which fall short in assessing the quality in open-ended scenarios.To this end, we introduce INTER-LEAVEDBENCH, the first benchmark carefully curated for the evaluation of interleaved textand-image generation.INTERLEAVEDBENCH features a rich array of tasks to cover diverse real-world use cases.In addition, we present INTERLEAVEDEVAL, a strong reference-free metric powered by GPT-4o to deliver accurate and explainable evaluation.We carefully define five essential evaluation aspects for IN-TERLEAVEDEVAL, including text quality, perceptual quality, image coherence, text-image coherence, and helpfulness, to ensure a comprehensive and fine-grained assessment.Through extensive experiments and rigorous human evaluation, we show that our benchmark and metric can effectively evaluate the existing models with a strong correlation with human judgments surpassing previous reference-based metrics.We also provide substantial findings and insights to foster future research in interleaved generation and its evaluation. 1

During conversations, the human flow of thoughts may result in topic shifts and evolution.In open-domain dialogue systems, it is crucial to track the topics discussed and recommend relevant topics to be included in responses to have effective conversations.Furthermore, topic evolution is needed to prevent stagnation as conversation length increases.Existing open-domain dialogue systems do not pay sufficient attention to topic evolution and shifting, resulting in performance degradation due to ineffective responses as conversation length increases.To address the shortcomings of existing approaches, we propose EVOLV-CONV.EVOLVCONV conducts real-time conversation topic and user preference tracking and utilizes the tracking information to evolve and shift topics depending on conversation status.We conduct extensive experiments to validate the topic evolving and shifting capabilities of EVOLVCONV as conversation length increases.Un-referenced evaluation metric UniEval compare EVOLVCONV with the baselines.Experimental results show that EVOLV-CONV maintains a smooth conversation flow without abruptly shifting topics; the probability of topic shifting ranges between 5%-8% throughout the conversation.EVOLVCONV recommends 4.77% more novel topics than the baselines, and the topic evolution follows balanced topic groupings.Furthermore, we conduct user surveys to test the practical viability of EVOLVCONV.User survey results reveal that responses generated by EVOLVCONV are preferred 47.8% of the time compared to the baselines and comes second to real human responses.

Procedural content generation (PCG) in video games offers unprecedented opportunities for customization and user engagement. Working within the specialized context of role-playing games (RPGs), we introduce a novel framework for quest and dialogue generation that places the player at the core of the generative process. Drawing on a hand-crafted knowledge base, our method grounds generated content with in-game context while simultaneously employing a large-scale language model to create fluent, unique, accompanying dialogue. Through human evaluation, we confirm that quests generated using this method can approach the performance of hand-crafted quests in terms of fluency, coherence, novelty, and creativity; demonstrate the enhancement to the player experience provided by greater dynamism; and provide a novel, automated metric for the relevance between quest and dialogue. We view our contribution as a critical step toward dynamic, co-creative narrative frameworks in which humans and AI systems jointly collaborate to create unique and user-specific playable experiences.