Alexander Spangher, Michael Lu, Sriya Kalyan, Hyundong Justin Cho, Tenghao Huang, Weiyan Shi, Jonathan May. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Information prioritization plays an important role in the way we perceive and understand the world.Homepage layouts, which are daily and manually curated by expert human news editors, serve as a tangible proxy for this prioritization.In this work, we present NewsHomepages, a novel and massive dataset of over 3,000 news website homepages, including local, national, and topic-specific outlets, captured twice daily over a five-year period.We develop a scalable pairwise preference model to capture ranked preferences between news items and confirm that these preferences are stable and learnable: our models infer editorial preference with over 0.7 F1 score (based on human trials).To demonstrate the importance of these learned preferences, we (1) perform a novel analysis showing that outlets across the political spectrum share surprising preference agreements and (2) apply our models to rankorder a collection of local city council policies passed over a ten-year period in San Francisco, assessing their "newsworthiness".Our findings lay the groundwork for leveraging implicit cues to deepen our understanding of human informational preference.

Alexander Spangher, Serdar Tumgoren, Ben Welsh, Nanyun Peng, Emilio Ferrara, Jonathan May. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Journalists engage in multiple steps in the news writing process that depend on human creativity, like exploring different "angles" (i.e. the specific perspectives a reporter takes).These can potentially be aided by large language models (LLMs).By affecting planning decisions, such interventions can have an outsize impact on creative output.We advocate a careful approach to evaluating these interventions to ensure alignment with human values.In a case study of journalistic coverage of press releases, we assemble a large dataset of 250k press releases 1 and 650k articles covering them. 2 We develop methods to identify news articles that challenge and contextualize press releases.Finally, we evaluate suggestions made by LLMs for these articles and compare these with decisions made by human journalists.Our findings are three-fold: (1) Human-written news articles that challenge and contextualize press releases more take more creative angles and use more informational sources.(2) LLMs align better with humans when recommending angles, compared with informational sources.(3) Both the angles and sources LLMs suggest are significantly less creative than humans.

Yufei Tian, Tenghao Huang, Miri Liu, Derek Jiang, Alexander Spangher, Muhao Chen, Jonathan May, Nanyun Peng. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

Classifier-Free Guidance (CFG) has recently emerged in as a lightweight technique to encourage prompt-adherence in generations, yet has not yet been successfully applied to language modeling. In this work, we demonstrate across a wide array of benchmarks that CFG can be used broadly as an inference-time technique in pure language modeling. We show that CFG (1) improves the performance of Pythia, GPT-2 and LLaMA-family models across: Q&A, reasoning, code generation, and machine translation, achieving SOTA on LAMBADA with LLaMA-7B over PaLM-540B; (2) brings improvements equivalent to a model with twice the parameter-count; (3) can stack alongside other inference-time methods like Chain-of-Thought and Self-Consistency, yielding further improvements in difficult tasks; (4) can be used to increase the faithfulness and coherence of assistants in challenging form-driven and content-driven prompts: in human evaluations we show a 75% preference for using CFG over baseline.

Alexander Spangher, Zihan Xue, Te-Lin Wu, Mark Hansen, Jonathan May. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

The ability to infer pre-and postconditions of an action is vital for comprehending complex instructions, and is essential for applications such as autonomous instruction-guided agents and assistive AI that supports humans to perform physical tasks.In this work, we propose a task dubbed action condition inference, which extracts mentions of preconditions and postconditions of actions in instructional manuals.We propose a weakly supervised approach utilizing automatically constructed large-scale training instances from online instructions, and curate a densely human-annotated and validated dataset to study how well the current NLP models do on the proposed task.We design two types of models differ by whether contextualized and global information is leveraged, as well as various combinations of heuristics to construct the weak supervisions.Our experiments show a >20% F1-score improvement with considering the entire instruction contexts and a > 6% F1-score benefit with the proposed heuristics.However, the best performing model is still well-behind human performance.1 standalone Heuristics Examples Descriptions Entity-Tracing & Coref.… Slice 500 grams of onions.… … Heat the pan with olive oil.… … Place them in the frying pan.… Precondition 1 Precondition 2The shared entities are pan and onions (linked via co-references to them).

News articles are driven by the informational sources journalists use in reporting.Modeling when, how and why sources get used together in stories can help us better understand the information we consume and even help journalists with the task of producing it.In this work, we take steps toward this goal by constructing the largest and widest-ranging annotated dataset, to date, of informational sources used in news writing.We first show that our dataset can be used to train high-performing models for information detection and source attribution.Then, we introduce a novel task, source prediction, to study the compositionality of sources in news articles -i.e.how they are chosen to complement each other.We show good modeling performance on this task, indicating that there is a pattern to the way different sources are used together in news storytelling.This insight opens the door for a focus on sources in narrative science (i.e.planningbased language generation) and computational journalism (i.e. a source-recommendation system to aid journalists writing stories).1

Te-Lin Wu, Alex Spangher, Pegah Alipoormolabashi, Marjorie Freedman, Ralph Weischedel, Nanyun Peng. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.

Alexander Spangher, Xiang Ren, Jonathan May, Nanyun Peng. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

As labeling schemas evolve over time, small differences can render datasets following older schemas unusable.This prevents researchers from building on top of previous annotation work and results in the existence, in discourse learning in particular, of many small classimbalanced datasets.In this work, we show that a multitask learning approach can combine discourse datasets from similar and diverse domains to improve discourse classification.We show an improvement of 4.9% Micro F1-score over current state-of-the-art benchmarks on the NewsDiscourse dataset, one of the largest discourse datasets recently published, due in part to label correlations across tasks, which improve performance for underrepresented classes.We also offer an extensive review of additional techniques proposed to address resource-poor problems in NLP, and show that none of these approaches can improve classification accuracy in our setting 1 .

The Russia-based Internet Research Agency (IRA) carried out a broad information campaign in the U.S. before and after the 2016 presidential election. The organization created an expansive set of internet properties: web domains, Facebook pages, and Twitter bots, which received traffic via purchased Facebook ads, tweets, and search engines indexing their domains. In this paper, we focus on IRA activities that received exposure through search engines, by joining data from Facebook and Twitter with logs from the Internet Explorer 11 and Edge browsers and the Bing.com search engine.