Abstract In many application domains, domain-specific languages can allow domain experts to contribute to collaborative projects more correctly and efficiently. To do so, they must be able to understand program structure from reading existing source code. With high-quality data becoming an increasingly important resource, the creation of data pipelines is an important application domain for domain-specific languages. We execute a mixed-method study consisting of a controlled experiment and a follow-up descriptive survey among the participants to understand the effects of a domain-specific language on bottom-up program understanding and generate hypotheses for future research. During the experiment, participants ( $$n=57$$ ) need the same time (Wilcoxon signed-rank test, $$W = 750$$ , $$p =.546$$ , RBC = .093) to solve program structure comprehension tasks, but submit significantly more correct solutions (McNemar’s test, $$\chi ^{2}_{1} = 11.17$$ , $$p = <.001$$ , OR = 4.8) when using the domain-specific language. In the descriptive survey, participants describe reasons related to the programming language itself, such as a better pipeline overview, more enforced code structure, and a closer alignment to the mental model of a data pipeline. In addition, human factors such as less required programming experience and the ability to reuse experience from other data engineering tools are discussed. Based on these results, domain-specific languages are a promising tool for creating data pipelines that can increase correct understanding of program structure and lower barriers to entry for domain experts. Open questions exist to make more informed implementation decisions for domain-specific languages for data pipelines in the future.

Microservices are a popular software architectural style that decomposes a large application into smaller services.These microservices integrate at runtime to deliver business value to the users.With an increasing number of microservices, software projects become more difficult to manage.Specifically, maintaining consistent and up-to-date documentation becomes a challenge that can significantly affect the integration efforts in such projects.In this article, we present MSAdoc, an open source tool that helps to prevent documentation from going out-of-date quickly.The tool (1) enables decentralized documentation close to the source code of each microservice and those who have to document it, (2) aggregates documentation centrally across individual microservices to make the documentation accessible in one place and generate higher-order documentation, while (3) supporting technological heterogeneity by relying on the technology-agnostic JSON format.Using a tool like MSAdoc that implements several best practices, practitioners can accommodate the decentralized nature of microservice-based projects and alleviate the problem of maintaining central documentation that quickly becomes outdated.

Abstract Microservices are a popular architectural style that allows systems to be built from a potentially large number of microservices, all of which can be developed independently and by their own teams. As a resulting benefit, development teams can choose the technologies optimal for their microservices, leading to a diversity of different programming languages, frameworks, and further technology in use. However, this heterogeneity presents challenges as it prevents code reuse and complicates moving individuals between microservices due to knowledge hurdles. We performed 15 expert interviews in a qualitative survey to build a theory on how technological heterogeneity can be balanced in microservice architectures to reach a context-dependent compromise between its benefits and drawbacks. We contribute by (1) gathering empirical data from industry professionals on a research topic that has been acknowledged but has only seen limited exploration so far, (2) developing a comprehensive theory of technology heterogeneity as a major integration challenge in microservice-based projects, (3) proposing a framework to overcome the challenge of balancing technological heterogeneity in microservice architectures, (4) optimizing the theory’s presentation for practical use in industry by using the well-known pattern format, and (5) generating research hypotheses to guide and inspire future investigations into this phenomenon.

Microservices have become an important architectural style for building robust and scalable software systems. A system’s functionality is split into independent units, the microservices, that communicate over a network and can be deployed independently. The shift of complexity into the integration layer necessitates enhanced collaboration among stakeholders, stressing the importance of effective communication. We aim to streamline communication between stakeholders in microservice-based projects by constructing a framework for enhanced clarity, a taxonomy, by answering our research question: “How can microservice integration techniques be classified?” We conducted a thematic analysis of literature and six expert interviews to identify microservice integration techniques and construct a taxonomy. The results of this study are (i) a taxonomy for microservice integration techniques consisting of five main and ten refined categories, (ii) the classification of 121 found integration techniques, (iii) an illustration of the taxonomy usage based on three selected techniques to demonstrate the procedure in case of classification ambiguity, (iv) a comparison of data gathered from literature with the interviews, and (v) comprehensive supplementary materials. The taxonomy offers a structured framework to classify microservice integration techniques and enhances the understanding of the diverse landscape of microservice integration techniques, including organizational ones that are often overlooked. Practitioners can discover integration techniques through the taxonomy and apply them with guidance provided in the supplementary materials. • A hierarchical taxonomy of microservice integration techniques was constructed through a thematic analysis of literature and expert interviews. • Main categories: within service, among services, with external systems. • The level of control over the integration counterpart implies the integration effort. • Organizational, procedural, and social aspects require further research. • Practitioners struggle with the steep learning curve of microservices.

ABSTRACT A large part of data science projects is spent on data engineering. Especially in open data contexts, data quality issues are prevalent and are often tackled by non‐professional programmers. We introduce and evaluate Jayvee, a domain‐specific language for data engineering aimed at reducing barriers to building data pipelines. We show that a structured DSL can have positive effects on speed, ease of use, and quality for data engineering by non‐professional developers. For this, we present an empirical quantitative study, in which we compare the performance of students as proxies for non‐professional programmers using Jayvee with Python and Pandas. We search for reasons for the empirical findings using a follow‐up interview study on how using a DSL changes how non‐professional programmers build data pipelines. Participants solve a subset of tasks faster, more easily, and with higher quality when using Jayvee compared to Python. Interviewees describe tradeoffs regarding the DSL's more limited features, stricter code structure, and explicit descriptions. Jayvee is found to be more approachable, which leads to a more guided development flow. New data engineering languages should provide good tooling and documentation, plan how to visualize intermediate data and consider new development workflows involving tools like ChatGPT to find adoption.