Abstract In many application domains, domain-specific languages can allow domain experts to contribute to collaborative projects more correctly and efficiently. To do so, they must be able to understand program structure from reading existing source code. With high-quality data becoming an increasingly important resource, the creation of data pipelines is an important application domain for domain-specific languages. We execute a mixed-method study consisting of a controlled experiment and a follow-up descriptive survey among the participants to understand the effects of a domain-specific language on bottom-up program understanding and generate hypotheses for future research. During the experiment, participants ( $$n=57$$ ) need the same time (Wilcoxon signed-rank test, $$W = 750$$ , $$p =.546$$ , RBC = .093) to solve program structure comprehension tasks, but submit significantly more correct solutions (McNemar’s test, $$\chi ^{2}_{1} = 11.17$$ , $$p = <.001$$ , OR = 4.8) when using the domain-specific language. In the descriptive survey, participants describe reasons related to the programming language itself, such as a better pipeline overview, more enforced code structure, and a closer alignment to the mental model of a data pipeline. In addition, human factors such as less required programming experience and the ability to reuse experience from other data engineering tools are discussed. Based on these results, domain-specific languages are a promising tool for creating data pipelines that can increase correct understanding of program structure and lower barriers to entry for domain experts. Open questions exist to make more informed implementation decisions for domain-specific languages for data pipelines in the future.

Abstract Microservices are a popular architectural style that allows systems to be built from a potentially large number of microservices, all of which can be developed independently and by their own teams. As a resulting benefit, development teams can choose the technologies optimal for their microservices, leading to a diversity of different programming languages, frameworks, and further technology in use. However, this heterogeneity presents challenges as it prevents code reuse and complicates moving individuals between microservices due to knowledge hurdles. We performed 15 expert interviews in a qualitative survey to build a theory on how technological heterogeneity can be balanced in microservice architectures to reach a context-dependent compromise between its benefits and drawbacks. We contribute by (1) gathering empirical data from industry professionals on a research topic that has been acknowledged but has only seen limited exploration so far, (2) developing a comprehensive theory of technology heterogeneity as a major integration challenge in microservice-based projects, (3) proposing a framework to overcome the challenge of balancing technological heterogeneity in microservice architectures, (4) optimizing the theory’s presentation for practical use in industry by using the well-known pattern format, and (5) generating research hypotheses to guide and inspire future investigations into this phenomenon.

ABSTRACT A large part of data science projects is spent on data engineering. Especially in open data contexts, data quality issues are prevalent and are often tackled by non‐professional programmers. We introduce and evaluate Jayvee, a domain‐specific language for data engineering aimed at reducing barriers to building data pipelines. We show that a structured DSL can have positive effects on speed, ease of use, and quality for data engineering by non‐professional developers. For this, we present an empirical quantitative study, in which we compare the performance of students as proxies for non‐professional programmers using Jayvee with Python and Pandas. We search for reasons for the empirical findings using a follow‐up interview study on how using a DSL changes how non‐professional programmers build data pipelines. Participants solve a subset of tasks faster, more easily, and with higher quality when using Jayvee compared to Python. Interviewees describe tradeoffs regarding the DSL's more limited features, stricter code structure, and explicit descriptions. Jayvee is found to be more approachable, which leads to a more guided development flow. New data engineering languages should provide good tooling and documentation, plan how to visualize intermediate data and consider new development workflows involving tools like ChatGPT to find adoption.