Multi-stage programming lets programmers write meta-programs that generate efficient code. Staging is typically realized either as a language primitive with quotations and splices (e.g., MetaML and its descendants), or as a library embedded in a host language (e.g., Lightweight Modular Staging). Unlike quotation-based approaches, practical library-based systems combine staged evaluation with automatic let-insertion to preserve evaluation order, along with optimizations that improve residual code. Despite their popularity and practical importance, this combination has received little semantic treatment, making it difficult to reason about correctness or to compare systematically with other staging paradigms. Using functional programming techniques, this pearl illuminates the operational aspects of staged evaluation with automatic let-insertion and optimizations as found in library-based staging systems. For a core two-stage language, we develop a series of definitional interpreters that concisely describe staged evaluation generating optimized, let-inserted residual programs. The interpreters are written in the extended continuation-passing style, where two continuations naturally account for let-insertion. With minor refactoring, we showcase a suite of optimizations, ranging from simple constant propagation/folding, common subexpression elimination, and dead-code elimination to more involved optimizations such as beta-inlining, partially-static data, and code motion. Each optimization is presented as a small, modular extension integrated into staged evaluation, requiring neither additional effort from the meta-programmer, nor complex post-hoc compiler infrastructure.

Dependency extraction based on static analysis lays the groundwork for a wide range of applications. However, dynamic language features in Python make code behaviors obscure and nondeterministic; consequently, it poses huge challenges for static analyses to resolve symbol-level dependencies. Although prosperous techniques and tools are adequately available, they still lack sufficient capabilities to handle object changes, first-class citizens, varying call sites, and library dependencies. To address the fundamental difficulty for dynamic languages, this work proposes an effective and practical method namely PyAnalyzer for dependency extraction. PyAnalyzer uniformly models functions, classes, and modules into first-class heap objects, propagating the dynamic changes of these objects and class inheritance. This manner better simulates dynamic features like duck typing, object changes, and first-class citizens, resulting in high recall results without compromising precision. Moreover, PyAnalyzer leverages optional type annotations as a shortcut to express varying call sites and resolve library dependencies on demand. We collected two micro-benchmarks (278 small programs), two macro-benchmarks (59 real-world applications), and 191 real-world projects (10MSLOC) for comprehensive comparisons with 7 advanced techniques (i.e., Understand, Sourcetrail, Depends, ENRE19, PySonar2, PyCG, and Type4Py). The results demonstrated that PyAnalyzer achieves a high recall and hence improves the F1 by 24.7% on average, at least 1.4x faster without an obvious compromise of memory efficiency. Our work will benefit diverse client applications.

Dependencies among software entities are the foundation for much of the research on software architecture analysis and architecture analysis tools. Dynamically typed languages, such as Python, JavaScript and Ruby, tolerate the lack of explicit type references, making certain dependencies indiscernible by a purely syntactic analysis of source code. We call thesepossible dependencies, in contrast with theexplicit dependenciesthat are directly manifested in source code. We find that existing architecture analysis tools have not taken possible dependencies into consideration. An important question therefore is:to what extent will these missing possible dependencies impact architecture analysis?To answer this question, we conducted a study of 499 open-source Python projects, employing type inference techniques and type hint practices to discern possible dependencies. We investigated the consequences of possible dependencies in three software maintenance contexts, including capturing co-change relations recorded in revision history, measuring architectural maintainability, and detecting architecture anti-patterns that violate design principles and impact maintainability. Our study revealed that the impact of possible dependencies on architecture-level maintainability is substantial—higher than that of explicit dependencies. Our findings suggest that architecture analysis and tools should take into account, assess, and highlight the impacts of possible dependencies caused by dynamic typing.

Dynamic programming languages have been embracing gradual typing, which supports optional type annotations in source code. Type-annotating a complex and long-lasting codebase is indeed a gradual and expensive process, where two issues have troubled developers. First, there is few guidance about how to implement type annotations due to the existence of non-trivial type practices; second, there is few guidance about which portion of a codebase should be type-annotated first. To address these issues, this paper investigates the patterns of non-trivial type-annotation practices and features of type-annotated code files. Our study detected six patterns of type-annotation practices, which involve recovering and expressing design concerns. Moreover, we revealed three complementary features of type-annotated files. Besides, we implemented a tool for studying optional typing practice. We suggest that: 1) design concerns should be considered to improve type annotation implementation by following at least six patterns; 2) files critical to software architecture could be type-annotated in priority. We believe these guidelines would promote a better type annotation practice for dynamic languages.