Abstract For this paper, we have identified a list of 32 characteristic symptoms that is often discussed during assessments, generated by two experienced assessors who had undertaken maturity assessments in more than 300 companies. We then use cognitive mapping and the “five whys” technique to look beyond the symptoms and reveal the underlying problems or causes of the problems. Following that, we evaluate our findings through the design and evaluation of a web‐based tool where users can score statements based on a formulation of the symptoms. This enables us to recommend to users the areas where they probably need to improve. We designed the tool in three learning cycles of design evaluation and ended up in a summative evaluation where we compared the outcome of using the website tool with a CMMI maturity assessment. We conclude that a systematic quest for symptoms coupled with scoring statements based on the symptoms can point to improvement areas. However, we conclude that doing so is no substitute for a maturity assessment; the scoring of statements cannot reveal the maturity of the organization, but it can quickly and easily point in a useful direction and provide recommendations for going in that direction.

Evaluation of design artefacts and design theories is a key activity in Design Science Research (DSR), as it provides feedback for further development and (if done correctly) assures the rigour of the research. However, the extant DSR literature provides insufficient guidance on evaluation to enable Design Science Researchers to effectively design and incorporate evaluation activities into a DSR project that can achieve DSR goals and objectives. To address this research gap, this research paper develops, explicates, and provides evidence for the utility of a Framework for Evaluation in Design Science (FEDS) together with a process to guide design science researchers in developing a strategy for evaluating the artefacts they develop within a DSR project. A FEDS strategy considers why, when, how, and what to evaluate. FEDS includes a two-dimensional characterisation of DSR evaluation episodes (particular evaluations), with one dimension being the functional purpose of the evaluation (formative or summative) and the other dimension being the paradigm of the evaluation (artificial or naturalistic). The FEDS evaluation design process is comprised of four steps: (1) explicate the goals of the evaluation, (2) choose the evaluation strategy or strategies, (3) determine the properties to evaluate, and (4) design the individual evaluation episode(s). The paper illustrates the framework with two examples and provides evidence of its utility via a naturalistic, summative evaluation through its use on an actual DSR project.

Abstract The ISO/IEC 33014.2013 standard on process improvement includes a core activity called ‘identify the overall change strategy’, which includes selecting a change strategy among the many available. This selection can be carried out using the ImprovAbility model in which there is a framework of how to select change strategy. The framework contains 10 different change strategies to choose from. But which ones are chosen in practice? To answer that, we have analyzed data from 134 assessments in 129 organizations that have used the framework. We give a ranking of strategies chosen, and we analyze how they adapt the change strategy to their specific conditions. We conclude that the most often recommended organizational change strategy is optionality followed by three other strategies: specialist‐driven, production‐organized, and learning‐driven. Copyright © 2015 John Wiley & Sons, Ltd.

While agility has recently attracted considerable attention in studies of systems development and management of information technology (IT), there is limited recognition within the information syst...