Podcast script generation requires LLMs to synthesize structured, context-grounded dialogue from diverse inputs, yet systematic evaluation resources for this task remain limited.To bridge this gap, we introduce PodBench, a benchmark comprising 800 samples with inputs up to 21K tokens and complex multi-speaker instructions.We propose a multifaceted evaluation framework that integrates quantitative constraints with LLM-based quality assessment.Extensive experiments reveal that while proprietary models generally excel, open-source models equipped with explicit reasoning demonstrate superior robustness in handling long contexts and multi-speaker coordination compared to standard baselines.However, our analysis uncovers a persistent divergence where high instruction following does not guarantee high content substance.PodBench offers a reproducible testbed to address these challenges in long-form, audio-centric script generation.The code and data are publicly accessible here.

Extensive research has been conducted to explore the capability of Large Language Models (LLMs) for table reasoning and has significantly improved the performance on existing benchmarks. However, tables and user questions in real-world applications are more complex and diverse, presenting an unignorable gap compared to the existing benchmarks. To fill the gap, we propose a Multi-scale spreadsheet benchmark with Meta operations for Table reasoning, named as MiMoTable. Specifically, MiMoTable incorporates two key features. First, the tables in MiMoTable are all spreadsheets used in real-world scenarios, which cover seven domains and contain different types. Second, we define a new criterion with six categories of meta operations for measuring the difficulty of each question in MiMoTable, simultaneously as a new perspective for measuring the difficulty of the existing benchmarks. Experimental results show that Claude-3.5-Sonnet achieves the best performance with 77.4% accuracy, indicating that there is still significant room to improve for LLMs on MiMoTable. Furthermore, we grade the difficulty of existing benchmarks according to our new criteria. Experiments have shown that the performance of LLMs decreases as the difficulty of benchmarks increases, thereby proving the effectiveness of our proposed new criterion.

Despite recent efforts to develop large language models with robust long-context capabilities, the lack of long-context benchmarks means that relatively little is known about their performance. To alleviate this gap, in this paper, we propose Counting-Stars, a multi-evidence, position-aware, and scalable benchmark designed to evaluate the multi-evidence retrieval capabilities of long-context LLMs. Counting-Stars comprises two counting-based multiple pieces of evidence retrieval tasks: searching and reasoning. Using Counting-Stars, we conducted experiments to evaluate several long-context LLMs, including GPT-4 Turbo, Gemini 1.5 Pro, Claude3 Opus, GLM-4, and Moonshot-v1. Extensive experimental results demonstrate that Gemini 1.5 Pro achieves the best overall results, while GPT-4 Turbo exhibits the most stable performance across various tasks. Furthermore, our analysis of these LLMs, which have been extended to handle long-context scenarios, indicates that significant room for improvement remains as the length of the input context and the complexity of the tasks increase.

Utilizing Large Language Models (LLMs) as evaluators to assess the performance of other LLMs has garnered attention. However, this evaluation approach is affected by potential biases within LLMs, raising concerns about the accuracy and reliability of the evaluation results of LLMs. To address this issue, we propose and explore two many-shot In-Context Learning (ICL) prompt templates to help LLM evaluators mitigate potential biases: Many-Shot with Reference (MSwR) and Many-Shot without Reference (MSoR). Specifically, the former utilizes in-context examples with model-generated rationales as references, while the latter does not include these references. Using these prompt designs, we investigate the impact of increasing the number of in-context examples on the consistency and quality of the evaluation results. Experimental results show that advanced LLMs, such as GPT-4, perform better in the many-shot regime than in the zero-shot regime. Furthermore, in most cases, MSwR performs significantly better than MSoR.

Although large-scale video-language pre-training models, which usually build a global alignment between the video and the text, have achieved remarkable progress on various downstream tasks, the idea of adopting fine-grained information during the pre-training stage is not well explored. In this work, we propose STOA-VLP, a pre-training framework that jointly models object and action information across spatial and temporal dimensions. More specifically, the model regards object trajectories across frames and multiple action features from the video as fine-grained features. Besides, We design two auxiliary tasks to better incorporate both kinds of information into the pre-training process of the video-language model. The first is the dynamic object-text alignment task, which builds a better connection between object trajectories and the relevant noun tokens. The second is the spatial-temporal action set prediction, which guides the model to generate consistent action features by predicting actions found in the text. Extensive experiments on three downstream tasks (video captioning, text-video retrieval, and video question answering) demonstrate the effectiveness of our proposed STOA-VLP (e.g. 3.7 Rouge-L improvements on MSR-VTT video captioning benchmark, 2.9% accuracy improvements on MSVD video question answering benchmark, compared to previous approaches).

Video-language pre-training has attracted considerable attention recently for its promising performance on various downstream tasks. Most existing methods utilize the modality-specific or modality-joint representation architectures for the cross-modality pre-training. Different from previous methods, this paper presents a novel architecture named Memory-augmented Inter-Modality Bridge (MemBridge), which uses the learnable intermediate modality representations as the bridge for the interaction between videos and language. Specifically, in the transformer-based cross-modality encoder, we introduce the learnable bridge tokens as the interaction approach, which means the video and language tokens can only perceive information from bridge tokens and themselves. Moreover, a memory bank is proposed to store abundant modality interaction information for adaptively generating bridge tokens according to different cases, enhancing the capacity and robustness of the inter-modality bridge. Through pre-training, MemBridge explicitly models the representations for more sufficient inter-modality interaction. Comprehensive experiments show that our approach achieves competitive performance with previous methods on various downstream tasks including video-text retrieval, video captioning, and video question answering on multiple datasets, demonstrating the effectiveness of the proposed method. The code has been available at https://github.com/jahhaoyang/MemBridge.

Video-language pre-training (VLP) has attracted increasing attention for cross-modality understanding tasks. To enhance visual representations, recent works attempt to adopt transformer-based architectures as video encoders. These works usually focus on the visual representations of the sampled frames. Compared with frame representations, frame patches incorporate more fine-grained spatio-temporal information, which could lead to a better understanding of video contents. However, how to exploit the spatio-temporal information within frame patches for VLP has been less investigated. In this work, we propose a method to learn tube tokens to model the key spatio-temporal information from frame patches. To this end, multiple semantic centers are introduced to focus on the underlying patterns of frame patches. Based on each semantic center, the spatio-temporal information within frame patches is integrated into a unique tube token. Complementary to frame representations, tube tokens provide detailed clues of video contents. Furthermore, to better align the generated tube tokens and the contents of descriptions, a local alignment mechanism is introduced. The experiments based on a variety of downstream tasks demonstrate the effectiveness of the proposed method.

Most methods tackle zero-shot video classification by aligning visual-semantic representations within seen classes, which limits generalization to unseen classes. To enhance model generalizability, this paper presents an end-to-end framework that preserves alignment and uniformity properties for representations on both seen and unseen classes. Specifically, we formulate a supervised contrastive loss to simultaneously align visual-semantic features (i.e., alignment) and encourage the learned features to distribute uniformly (i.e., uniformity). Unlike existing methods that only consider the alignment, we propose uniformity to preserve maximal-info of existing features, which improves the probability that unobserved features fall around observed data. Further, we synthesize features of unseen classes by proposing a class generator that interpolates and extrapolates the features of seen classes. Besides, we introduce two metrics, closeness and dispersion, to quantify the two properties and serve as new measurements of model generalirability. Experiments show that our method significantly outperforms SoTA by relative improvements of 28.1% on UCF101 and 27.0% on HMDB51. Code is available11https://github.com/ShipuLoveMili/CVPR2022-AURL.

In this study, we extend traditional (single-target) hybrid systems to multi-target hybrid systems with a focus on the multi-maneuvering-target tracking system. This system consists of a continuous state, a discrete and switchable state, and a discrete, time-constant, and unique state. By defining a new generalized labeled multi-Bernoulli density, we prove that it is closed under the Chapman-Kolmogorov prediction and Bayes update for multi-target hybrid systems. In other words, we provide the exact derivation of a solution to this system, i.e., the multi-model generalized labeled multi-Bernoulli filter, which has been developed without strict proof.

Computer science is in increasing demand at the K-12 level. However, due to a lack of qualified educators, numerous programs (e.g., Google's CS4HS, NSF's CSForAll) have been created to train teachers and increase student access to computer science. Although these programs have seen some success, schools in areas with lower population density continue to struggle to offer computer science opportunities. These schools face unique challenges due to their low population density, including increasing teaching demands in core subjects, smaller teaching staff, and a smaller pipeline of students, making it difficult to justify offering a course in computer science. This poster draws on our experience of collaborating with K-12 teachers in rural schools in western Wisconsin to help mitigate these unique challenges. We have obtained two Google CS4HS grants to hold educational computer science workshops with teachers from across the region. These workshops allowed us to work closely with teachers and help them become more comfortable with incorporating computer science into their classrooms, while also helping us to form relationships with both teachers and administrators in a number of districts. Building on these relationships, we are taking the next step of working more closely with individual school districts to form professional learning communities. Our goal in this work is to connect K-12 teachers within each district and help facilitate a robust pipeline of students interested in computer science and justify offering AP CS courses at the high school level.

Muyu Zhang, Bing Qin, Mao Zheng, Graeme Hirst, Ting Liu. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2015.

Agent-based software engineering combines the strengths of agent theory with the practical software engineering methods for developing large intelligent systems. Designing and implementing agent-based systems are complicated tasks because the agent's BDI characteristics (beliefs, desires, intentions) when codified in the design/implementation as software characteristics may have acquired different connotations. Consequently testing the reliability of the implementation is equally hard. It is necessary to choose an agent's software abstraction that has a minimal deviation from the BDI agent concept, and use it as a formal basis to test an implementation. This paper proposes one such method for conformance testing of BDI properties in an agent-based system.

Real-time reactive systems are often safety-critical and hence must be thoroughly analyzed before they are deployed. We discuss methods for generating tests from object-oriented specifications of real-time reactive systems. Formal descriptions contain information on system entities, as mandated by requirements, as well as environmental entities, included in a domain model. These are exploited in the testing of an implementation, either for generating and sequencing tests, or as an oracle in evaluating test results. A tool has been developed based on the techniques and is being applied to several case studies developed in TROMLAB, a framework for a rigorous development of real-time reactive systems. The train-gate-controller (TGC) case study presented is one of them.

Real-time reactive systems are complex systems to design and verify. Rigorous testing of real-time reactive systems complement the more difficult and expensive formal verification process. This paper discusses a rigorous method for block-box testing of real-time reactive systems, whose design specifications are given in the timed reactive object model (TROM) formalism.

The paper discusses complexity categories associated with the phases of a process model that is followed in a rigorous development of real time reactive systems. We describe the complexity that is inherent in each phase and give measures to quantify it and methods to contain it. One of the goals of the work is to automate the measurement methods and integrate them into TROMLAB, a rigorous framework for the development of real time reactive systems.