Difference-in-differences (DiD) has been extensively applied in the literature to elicit the average causal effect of an intervention or policy. Though researchers explore heterogeneity in the treatment effect with respect to time or some observed covariate (usually driven by domain knowledge), there is limited work on a principled, algorithmic approach to discovering the subgroups that exhibit heterogeneity in the treatment effect in DiD settings. In this research, we propose the Difference-in-Differences Subset Scan (DiD-Scan), which finds subregions of the observed covariate space corresponding to anomalous patterns in the conditional mean treatment effect. We mold our method to the DiD setting by also enabling a scan for dynamic effects over time and across multiple outcome variables. This supports the discovery of patterns where a subset of outcomes are highly impacted for a limited time window. We develop a generalized likelihood ratio-based scoring function to quantify the treatment effect of a given subgroup and propose a computationally efficient method to discover the subgroups with the largest treatment effect. We extend the method to consider correlations across time, a common condition in difference-in-difference settings that increases the difficulty of subset identification. Lastly, we demonstrate the efficacy and interpretability of the method with both simulations and applications to real datasets, replicating and extending two published studies.

The use of machine learning (ML) models to assess and score textual data has become increasingly pervasive in an array of contexts including natural language processing, information retrieval, search and recommendation, and credibility assessment of online content. A significant disruption at the intersection of ML and text are text-generating large-language models (LLMs) such as generative pre-trained transformers (GPTs). We empirically assess the differences in how ML-based scoring models trained on human content assess the quality of content generated by humans versus GPTs. To do so, we propose an analysis framework that encompasses essay scoring ML models, human- and ML-generated essays, and a statistical model that parsimoniously considers the impact of type of respondent, prompt genre, and the ML model used for assessment model. A rich testbed is utilized that encompasses 18,460 human-generated and GPT-based essays. Results of our benchmark analysis reveal that LLMs and transformer pretrained language models (PLMs) more accurately score human essay quality as compared to CNN/RNN and feature-based ML methods. Interestingly, we find that LLMs and transformer PLMs tend to score GPT-generated text 10–20% higher on average, relative to human-authored documents. Conversely, traditional deep learning and feature-based ML models score human text considerably higher. Further analysis reveals that even though the LLMs and transformer PLMs are exclusively fine-tuned on human text, they more prominently attend to certain tokens appearing only in GPT-generated text, possibly (in part) due to familiarity/overlap in pre-training. Our framework and results have implications for text classification settings where automated scoring of text is likely to be disrupted by generative AI.