Warning: This article contains stereotypical and offensive content.

Patent examination remains an ongoing challenge in the NLP literature even after the advent of large language models (LLMs), as it requires an extensive yet nuanced human judgment on whether a submitted $\textit{claim}$ meets the statutory standards of $\textit{novelty}$ and $\textit{non-obviousness}$ against previously granted claims—$\textit{prior art}$—in expert domains. Previous NLP studies have approached this challenge as a prediction task (e.g., forecasting grant outcomes) with high-level proxies such as similarity metrics or classifiers trained on historical labels. However, this approach often overlooks the step-by-step evaluations that examiners must make with profound information, including rationales for the decisions provided in $\textit{office actions}$ documents, which also makes it harder to measure the current state of techniques in patent review processes. To fill this gap, we construct PANORAMA, a dataset of 8,143 U.S. patent examination records that preserves the full decision trails, including original applications, all cited references, $\textit{Non-Final Rejections}$, and $\textit{Notices of Allowance}$. Also, PANORAMA decomposes the trails into sequential benchmarks that emulate patent professionals' patent review processes and allow researchers to examine large language models' capabilities at each step of them. Our findings indicate that, although LLMs are relatively effective at retrieving relevant prior art and pinpointing the pertinent paragraphs, they struggle to assess the novelty and non-obviousness of patent claims. We discuss these results and argue that advancing NLP, including LLMs, in the patent domain requires a deeper understanding of real-world patent examination. Our dataset is openly available at https://huggingface.co/datasets/LG-AI-Research/PANORAMA.

Participatory Design (PD) is a commonly used method in Western cultures with roots in Scandinavia. After the COVID-19 pandemic, many PD practitioners have transitioned to an online space, where researchers and participants from different geographical locations can co-design together. Yet, we do not fully understand how different cultures collide, mix and blend in the online co-design space. By understanding how one’s culture influences the participation of a child, researchers can better structure sessions to elicit diverse input. This study examines a case study of researchers and participants from South Korea and the United States co-designing together. Close analysis of video recordings, analytic memos, and parent/child interview data demonstrates the ways in which children participated in design sessions. Facilitators paid special attention to various cultural sensitivities. Applying the theory of cultural awareness from public health literature, we offer implications on how scholars who are not fully embedded in the culture where co-design happens can understand other cultural norms and in response, create spaces in which co-design can occur successfully despite different cultural norms. Our findings show how different parts of the ecological systems from ideology, education systems, and ethnicity all influence children’s participation in sessions.

Would you rather go 1000 days without the Internet or five days where anyone can read your mind? We present “Would You Rather” (WYR), a technique for generating formative design insights (inspired by the conversational game of the same name) that combines design provocations with forced-choice scaffolding. Here, we describe the components of a WYR session, which include scenario generation, voting, and group discussion. As children disproportionately benefit from scaffolding during the co-design process, we also report on an evaluation of the technique with 16 children, conducted across seven sessions and spanning the course of one year. We find that WYR fulfills recommendations for focus groups (e.g. eliciting mental models and values, producing focused yet animated discussion) and leverages playfulness, humor, structure, and forced choice to overcome known common challenges of designing with children.

Co-designing with children in an online environment is increasingly important due to external factors, such as the COVID-19 pandemic, and the diversification and inclusion of youth participants. Many prior studies about co-design with youth focus on co-located or asynchronous online sessions. However, conducting synchronous online co-design sessions adds layers of complexity and uncertainty to collaboration. This paper introduces a model explicating factors to consider when co-designing with children synchronously in an online space. We examined ten consecutive intergenerational participatory design sessions online where children (ages 7-11) and adults designed new technologies. Along with highlighting unexpected moments and interactions, we use theories of improvisation to guide our understanding of dynamic situations that are out of the control of researchers. This work contributes to improving theoretical understanding of improvisation as a method of inquiry for co-designing with youth, and offers practical suggestions for suitable online co-design techniques and implementation.