Hyungyu Shin, Jingyu Tang, Yoonjoo Lee, Nayoung Kim, Hyunseung Lim, Ji Yong Cho, Hwajung Hong, Moontae Lee, Juho Kim. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, Sue Hyun Park, Hyeonbin Hwang, Jinkyung Jo, Hyowon Cho, Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, Se June Joo, Miyoung Ko, Yoonjoo Lee, Hyungjoo Chae, Jamin Shin, Joel Jang, Seonghyeon Ye, Bill Yuchen Lin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, Minjoon Seo. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

Patent examination remains an ongoing challenge in the NLP literature even after the advent of large language models (LLMs), as it requires an extensive yet nuanced human judgment on whether a submitted $\textit{claim}$ meets the statutory standards of $\textit{novelty}$ and $\textit{non-obviousness}$ against previously granted claims—$\textit{prior art}$—in expert domains. Previous NLP studies have approached this challenge as a prediction task (e.g., forecasting grant outcomes) with high-level proxies such as similarity metrics or classifiers trained on historical labels. However, this approach often overlooks the step-by-step evaluations that examiners must make with profound information, including rationales for the decisions provided in $\textit{office actions}$ documents, which also makes it harder to measure the current state of techniques in patent review processes. To fill this gap, we construct PANORAMA, a dataset of 8,143 U.S. patent examination records that preserves the full decision trails, including original applications, all cited references, $\textit{Non-Final Rejections}$, and $\textit{Notices of Allowance}$. Also, PANORAMA decomposes the trails into sequential benchmarks that emulate patent professionals' patent review processes and allow researchers to examine large language models' capabilities at each step of them. Our findings indicate that, although LLMs are relatively effective at retrieving relevant prior art and pinpointing the pertinent paragraphs, they struggle to assess the novelty and non-obviousness of patent claims. We discuss these results and argue that advancing NLP, including LLMs, in the patent domain requires a deeper understanding of real-world patent examination. Our dataset is openly available at https://huggingface.co/datasets/LG-AI-Research/PANORAMA.

Research promotion enables researchers to share advanced knowledge with pertinent academic communities. The question-and-answer (QA) style articles are effective for researchers to promote their research by enabling readers to understand research on complex subjects. Recent advances in large language models (LLMs) have opened avenues for supporting researchers in creating QA-style articles for research promotion. However, without the authors’ involvement, these models may only partially capture the researcher’s intention and voice. We developed AQUA, a research probe that enables researchers to co-create QA-style articles with LLMs to promote their research papers. A user study (n=12) reveals that LLMs reduced authors’ burden and helped them understand the readers’ perspectives. Nevertheless, LLMs failed to capture the unique intent of the authors, and their automated generation discouraged authors from carefully revising their answers. Based on our findings, we discuss human-LLM interaction design to enable authors to create QA-style articles that reflect their intention.

Planning is a self-regulated learning strategy and widely used behavior change technique that can help learners achieve academic goals (e.g., pass an exam, apply to college, or complete an online course). Numerous studies have tested the effects of planning interventions, but few have examined the content of learners’ plans and how it relates to their academic outcomes. Building on a large-scale intervention study, we conducted a qualitative content analysis of 650 learner plans sampled from 15 massive open online courses (MOOCs). We identified a number of planning tactics, compared their prevalence, and examined which ones significantly predict course progress and completion using regression analyses. We found that learners whose plans specify a time of day (e.g., morning, afternoon, night) are significantly more likely to complete a MOOC, but only 25% of the learners in our sample used this tactic. The high degree of variation in the effectiveness of planning tactics may contribute to mixed intervention findings in scale-up studies. Models of plan effectiveness can be used to provide feedback on the quality of learners’ plans and encourage them to use effective tactics to achieve their learning goals.

Online courses lower geographic barriers to educational access and attract learners from around the world. The resulting cultural diversity in online courses has implications for learning preferences, behaviors and outcomes, but established measures of culture are not adapted to educational contexts. We adapted and tested a survey instrument of cultural dimensions of learning that is grounded in cultural psychology research and spans four dimensions: knowledge construction, pedagogical orientation, uncertainty tolerance, and consensus building. We collected 600 responses in two online courses, conducted an explanatory factor analysis, and compared responses across five countries. We found that the instrument has a clear factor structure with high internal consistency, and it can distinguish cultures between countries. The instrument can be used to better understand learners and their culture in the process of course design and evaluation.

Public health research has developed a deep understanding of ways to help people live healthier lives through scalable interventions that change their behaviors. This work offers valuable insights for supporting learners in educational contexts, especially for improving self-regulation and goal-directed behaviors like completing a course of study--a persistent issue in formal and information post-secondary education. We present the widely adopted Behavior Change Technique (BCT) taxonomy as a model for systematically cataloging interventions in education and as a resource for inspiring new interventions in education based on public health evidence. Approaching the issue of learner attrition from the BCT perspective, we show how recent educational interventions fit into the BCT taxonomy and how the taxonomy can be used to develop new evidence-based intervention approaches. Borrowing insights from decades of public health research can advance parallel efforts in education to help learners at scale to stay on track and reach their academic goals.

Educators and researchers in online education have grappled with not only how to increase course completion but also how to make a broader impact that goes beyond online courses, such as course participants' real-world applications of the learned knowledge and skills. Research in social psychology and behavioral science suggests that social norms interventions, which convey norms shared in the community that people belong in to promote desirable behaviors, can offer a low-cost and scalable approach to encourage actions beyond the courses (ABCs). We tested three social norm interventions that presented a weekly normative message (descriptive, dynamic, or injunctive norm) with aggregate information about course participants' ABCs in the prior week. Randomized experiments in three online courses found effects on ABCs to be weak and moderated by norm message type and the complexity of the target behavior. Although the interventions did not improve course completion, the dynamic norm message was more effective at promoting ABCs for complex behaviors, such as developing environmental education activities.

University courses around the world suddenly transitioned to emergency remote instruction in response to the COVID-19 pandemic. We study changes in students' experience of support from their instructors and peers in large lecture courses. Social support can act as an important resource for students and buffer against mental distress. We find that students experienced more support from instructors but less support from their peers after the transition to remote instruction. Remote learning was less active and involved fewer peer interactions, with synchronous classes resembling online office hours and students struggling to get help. Our findings suggest the need for additional resources to help students stay connected and facilitate collaboration online.

The links between music and computation date back at least three decades. This trend has led to the development of learning environments for novices to make music and learn computational concepts. However, research connecting music and computation is mostly situated within the context of on-screen programming, with little research involving hybrid, tangible environments. Electronic textiles (e-textiles) is one such hybrid context where learners craft interactive physical artifacts by sewing microcontrollers, sensors, and actuators onto toys, clothes, or other fabric accessories. Prior studies show that Arduino-based e-textiles afford opportunities for students to learn basic concepts such as variables and control flow but not more advanced concepts such as arrays and for loops. In our study, we expand learning with e-textiles as learners compose and code music to address the following research question: What are the affordances of designing musical wearables to deepen computational conceptual learning? We present a case study of a group of four 15-17-year-old youth and an adult co-making a musical wearable-JazzHands-a glove augmented with actuators such as LEDs and speaker, and motion, touch, and light sensors. We analyzed videos of the group's collaborative design sessions, student daily journals, multiple versions of the code, and post-workshop, artifact-based interviews with the youth. Our analysis revealed that making musical wearables not only allowed the youth to express themselves and relate to computing as a creative endeavor but also afforded unique opportunities to work with constructs such as arrays indices, sensor data integration, and logical expressions to control for loops.

K-8 mathematics students must learn many procedures, such as addition and subtraction. Students frequently learn "buggy' variations of these procedures, which we ideally could identify automatically. This is challenging because there are many possible variations that reflect deep compositions of procedural thought. Existing approaches for K-8 math use manually specified variations which do not scale to new math algorithms or previously unseen misconceptions. Our system examines students' answers and infers how they incorrectly combine basic skills into complex procedures. We evaluate this approach on data from approximately 300 students. Our system replicates 86% of the answers that contain clear systematic mistakes (13%). Investigating further, we found 77% at least partially replicate a known misconception, with 53% matching exactly. We also present data from 29 participants showing that our system can demonstrate inferred incorrect procedures to an educator as successfully as a human expert.