Data science projects have become commonplace over the last decade. During this time, the practices of running such projects, together with the tools used to run them, have evolved considerably. Furthermore, there are various studies on data science workflows and data science project teams. However, studies looking into both workflows and teams are still scarce and comprehensive works to build a holistic view do not exist. This study bases on a prior case study on roles and processes in data science. The goal here is to create a deeper understanding of data science projects and development processes. We conducted a survey targeted at experts working in the field of data science (n=50) to understand data science projects’ team structure, roles in the teams, utilized project management practices and the challenges in data science work. Results show little difference between big data projects and other data science. The found differences, however, give pointers for future research on how agile data science projects are, and how important is the role of supporting project management personnel. The current study is work in progress and attempts to spark discussion and new research directions.

In the context of business intelligence, data warehousing is often perceived as an integral component of concrete business intelligence solutions. Since the nature of a traditional data warehouse is accumulative – data from operational systems is fed in to the system when it is ready for inclusion – and as the data from different component systems is interrelated, operations involving data warehouses have been traditionally considered tedious and delicate. Distinct steps take place one after another in a predefined, next to unalterable sequence. In this paper, we present an alternative model for dealing with data warehouses, where the goal is to apply principles of continuous software engineering in the domain of business intelligence. To validate the methodology, we present a tool chain that has been used in a real-life implementation of a business intelligence solution, together with experiences from its operations.

Software development projects have increasingly been adopting new practices, such as continuous delivery and deployment to enable rapid delivery of new features to end users. Tools that are commonly utilized with these practices generate a vast amount of data concerning various development events. Analysis of the data provides a lightweight data driven view on the software process. We present an efficient way of visualizing software process data to provide a good overall view on the features and potential problems of the process. We use the visualization in a case project that has become more agile by applying continuous integration and delivery together with development and infrastructure automation. We compare data visualizations with information gathered from the development team and describe how the evolution can be understood through our visualizations. The case project is a good example of how a change from a traditional long cycle development to a rapid cycle DevOps culture can actually be made in a few years. However, the results show that the team has to focus on the process improvement continuously in order to maintain continuous delivery all the time. As the main contribution, we present a lightweight way to software process visualization. Moreover, we discuss how such a heuristic can be used to track the characteristics of the target process.

Data warehousing is a process of integrating multiple data sources into one for, e.g., reporting purposes. An emerging modeling technique for this is the data vault method. The use of data vault creates many structurally similar data processing modifications in the transform phase of ETL work. Is it possible to automate the creation of transformations? Based on our study, the answer is mostly affirmative. Data vault modeling creates certain constraints to data warehouse entities. These model constraints and data vault table populating principles can be used to generate transformation code. Based on the original relational database model and data flow metadata we can gather populating principles. These can then be used to create general templates for each entity. Nevertheless, we need to note that the use of data flow metadata can be only partially automated and includes the only manual work phases in the process. In the end we can generate the actual transformation code automatically. In this paper, we carefully describe the creation of automation procedure and analyze the practical problems based on our experiences on PL/SQL proof of concept implementation. To the best of our knowledge, similar has not yet been described in the scientific literature.

In addition to functions, the architecture of a software system defines numerous other properties, commonly referred to as non-functional or quality properties. In web applications, two well-known architectural styles are common. They are resource-oriented architectural style, implemented using RESTful principles, and message-passing architectural style, implemented in XMPP. These architectural styles have their different benefits, whereas in real applications properties of both are sometimes needed. In this paper we discuss how RESTful web architecture can be complemented with XMPP. As an example, we use a distributed content management system, which is built using RESTful design guidelines and complemented with message passing architectural style implemented on XMPP. Features of both architectural styles are used in a fashion where no negative feature interaction takes place. In the completed system, both architectural styles are clearly identifiable, and their non-functional properties are preserved.