Context and motivation: There are several educational motivations for developing question banks with multiple variants of the same or similar task. One motivation might be that students can take tests at different times, or they are allowed several retakes of the same test. For instance, courses using self-paced mastery learning will typically have this feature, needing many variants of tests to avoid that students simply memorize answers. At the Norwegian University of Science and Technology, an introductory programming course for STEM teacher students is run according to a mastery learning paradigm. To mitigate test anxiety among students, a high level of test transparency is strived for with informal exercise tests that the students can use during test preparation, randomly pulling questions from the same question banks as used by the summative tests. After the 2023 run of the course the teacher found some tasks to have variants of unequal difficulty, which might be unfair in summative tests, and tried to fix this for the 2024 run of the course. Question / problem: This paper investigates whether there was a significant difference in difficulty between variants of the tasks, what were the causes for these differences, and whether the changes done between the first and second run of the course contributed to reducing the issues of unfairness. Principal ideas / results: This is done by analyzing the formative and summative test results for students who had consented to have their performance data used for research. Statistical analysis showed significant differences in difficulty for some variant groups, and the changes done for the second run of the course were indeed successful in reducing the variation in difficulty. Contribution: The paper analyses possible causes for the variation in difficulty, related to question framing, phrasing of answer options, and sequencing of code lines. The main contribution is to show that even minor changes to tasks can cause differences in performance, indicating that improvements to the task could reduce unfairness between variants. The paper concludes by identifying some factors to consider when designing tasks with many variants.

The Covid-19 pandemic led to increased use of home exams, with a perceived increase of cheating. Assessment integrity is a key challenge for higher education. Apart from remote proctoring, what other mitigations may be possible against cheating in home exams, and specifically for programming courses with huge classes? The paper presents our approaches to mitigate cheating, for CS1 based on questions with subtly different variants, for CS2 based on plagiarism detection and timestamps – in sufficient detail that others could use a similar approach. These two approaches can be partially effective against collaboration, but less so against contract cheating where help is acquired from an outside third party. Hence, towards the end of the paper we also outline possible approaches to mitigate such cheating, without or in addition to remote proctoring.

Software security is a complex topic, and for development projects it can be challenging to assess what security is necessary and cost-effective. Agile Software Development (ASD) values self-management. Thus, teams and their Product Owners are expected to also manage software security prioritisation. In this paper we build on the notion that security experts who want to influence the priority given to security in ASD need to do this through interactions and support for teams rather than prescribing certain activities or priorities. But to do this effectively, there is a need to understand what hinders and supports teams in prioritising security. Based on a longitudinal case study, this article offers insight into the strategy used by one security professional in an SME to influence the priority of security in software development projects in the company. The main result is a model of influences on security prioritisation that can assist in understanding what supports or hinders the prioritisation of security in ASD, thus providing recommendations for security professionals. Two alternative strategies are outlined for software security in ASD – prescribed and emerging – where we hypothesise that an emerging approach can be more relevant for SMEs doing ASD, and that this can impact how such companies should consider software security maturity.

A proper assessment of potential cyber threats is vital for security decision-making. This becomes an even more challenging task when dealing with new system designs and industry sectors where there is little or no historical data about past security incidents. We have developed a threat likelihood estimation approach that supports risk management under such circumstances. Quantifiable conditions are determined from the environment in which the system will reside and operate, that is the availability of potential threat actors, their opportunities of performing attacks, the required means that are needed for the attack to succeed, and motivation factors. Our research method follows the principles of practice research where both researchers and practitioners have played central roles in a real-life development project for a maritime communication system. We used a qualitative case study for feature-based evaluation of the approach and associated tool template, and to gather evidence on practical aspects such as suitability for purpose, efficiency and drawbacks from five user groups. The results show that representative participants from the cyber security and maritime community gave positive and consistent scores on the features, and regarded time usage, traceability of the threat assessment and the ability to indicate underlying uncertainty to be very appropriate. The approach has been proven useful for this domain and should be applicable to others as well, but the template requires up-front investments in gathering knowledge that is relevant and reusable in additional context situations.

As the field of computing education grows and matures, it has become essential to unite computing education and higher education research. Educational research has highlighted that how students study is crucial to their learning progress, and study behaviors have been found to play an important role in students’ academic success. This article presents the main results of a systematic literature review intended to determine what we know about the study behaviors of computing students and the role of educational design in shaping them. A taxonomy of study behaviors was developed and used to clarify and classify the definitions of study behavior, process, strategies, habits, and tactics as well as to identify their relationship to the educational context. The literature search resulted in 107 included papers, which were analyzed according to defined criteria and variables. The review of study behavior terminology found that the same terms are used to describe substantially different study behaviors, and the lack of standard terminology makes it difficult to compare findings from different papers. Furthermore, it was more common for papers to use study behaviors to explain other aspects of students rather than exploring and understanding them. Additionally, the results revealed a tendency to focus on specific educational contexts, predominantly introductory programming courses. Although computing education as a field is well equipped to expand the knowledge about both study behaviors and their connection to the educational context, the lack of common terminology and theories limits the impact. The taxonomy of study behaviors in computing education proposed in this article can contribute to contextualizing the research in such a way that researchers and educators across institutional borders can compare and utilize results. Last, the article outlines some areas for future research and recommendations for practice.

This Research Full Paper investigates the relationship between code writing tasks and other tasks like program comprehension and completion. In the context of exams, it is interesting to know whether auto-gradable comprehension and completion tasks can to some extent replace harder-to-grade code writing tasks and still test the same learning outcomes. This paper reports on the post mortem analysis of the 2019 e-exam in a university level programming course. Questions were analyzed for difficulty level, correlation, and discrimination. Program puzzles were found to correlate well with code writing, which is in line with previous findings by others. On average, code writing questions had higher difficulty, discrimination, and correlation with the total score than the other question formats - but not uniformly so. A more fine granular analysis of performance on individual interaction items sheds light on reasons why some questions worked better than others, with implications for future question design.

Ransomware is an epidemic that adversely affects the lives of both individuals and large companies, where criminals demand payments to release infected digital assets. In the wake of the ransomware success, Ransomware-as-a-Service (RaaS) has become a franchise offered through darknet marketplaces, allowing aspiring cybercriminals to take part in this dubious economy. We have studied contemporary darknet markets and forums over a period of two years using a netnographic research approach. Our findings show that RaaS currently seems like a modest threat relative to popular opinion. Compared to other types of illegal digital goods, there are rather few RaaS items offered for sale in darknet marketplaces, often with questionable authenticity. From our data we have created a value chain and descriptions of the actors involved in this economy.

Purpose Within critical-infrastructure industries, bow-tie analysis is an established way of eliciting requirements for safety and reliability concerns. Because of the ever-increasing digitalisation and coupling between the cyber and physical world, security has become an additional concern in these industries. The purpose of this paper is to evaluate how well bow-tie analysis performs in the context of security, and the study’s hypothesis is that the bow-tie notation has a suitable expressiveness for security and safety. Design/methodology/approach This study uses a formal, controlled quasi-experiment on two sample populations – security experts and security graduate students – working on the same case. As a basis for comparison, the authors used a similar experiment with misuse case analysis, a well-known technique for graphical security modelling. Findings The results show that the collective group of graduate students, inexperienced in security modelling, perform similarly as security experts in a well-defined scope and familiar target system/situation. The students showed great creativity, covering most of the same threats and consequences as the experts identified and discovering additional ones. One notable difference was that these naïve professionals tend to focus on preventive barriers, leading to requirements for risk mitigation or avoidance, while experienced professionals seem to balance this more with reactive barriers and requirements for incident management. Originality/value Our results are useful in areas where we need to evaluate safety and security concerns together, especially for domains that have experience in health, safety and environmental hazards, but now need to expand this with cybersecurity as well.

Educators and researchers have been working to understand the reasons that may be contributing to high dropout rates, and low rates of participation, by females in the computer and information sciences discipline. Along the same lines, and propelled by the increased need for information technology (IT) professionals worldwide, we implemented a students' survey during the fall of 2015 in Norway's primary university for technological education. In this initiative we aim to identify reasons that may be contributing to high dropout rates, low rates of participation by females and aspects important for the efficient preparation of young people for careers in computer science and information technology. The results provide valuable insights and allow us to take appropriate measures for enhancing students' learning experience in the computer and information sciences.

Model driven security has become an active area of research during the past decade. While many research works have contributed significantly to this objective by extending popular modeling notations to model security aspects, there has been little modeling support for state-based views of security issues. This paper undertakes a scientific approach to propose a new notational set that extends the UML (Unified Modeling Language) statecharts notation. An online industrial survey was conducted to measure the perceptions of the new notation with respect to its semantic transparency as well as its coverage of modeling state based security aspects. The survey results indicate that the new notation encompasses the set of semantics required in a state based security modeling language and was largely intuitive to use and understand provided very little training. A subject-based empirical evaluation using software engineering professionals was also conducted to evaluate the cognitive effectiveness of the proposed notation. The main finding was that the new notation is cognitively more effective than the original notational set of UML statecharts as it allowed the subjects to read models created using the new notation much quicker.

Security requirements elicitation and modelling are integral for the successful development of secure systems. However, there are a lot of similar yet not identical approaches that currently exist for security requirements modelling, which is confusing for researchers and practitioners hence some characterisation will be useful to give a better overview and understanding of advantages and disadvantages of various approaches. This paper provides a comparative review of i*-based and use case - based security modelling initiatives, using a characterisation framework with several dimensions. Our findings show that both categories of initiatives have significant conceptual similarities in the aspect of modelling language and method process, and coverage of security requirements modelling notions. They have conceptual differences in the aspect of: representation perspective, kind of security requirements engineering activities that are supported, the quality of specification that is generated and the specification techniques used, and the degree of support for software evolution.

The idea of security aware system development from the start of the engineering process is generally accepted nowadays and is becoming applied in practice. Many recent initiatives support this idea with special focus on security requirements elicitation. However, there are so far no techniques that provide integrated overviews of security threats and system architecture. One way to achieve this is by combining misuse cases with use case maps into misuse case maps (MUCM). This paper presents an experimental evaluation of MUCM diagrams focusing on identification of vulnerabilities and mitigations. The controlled experiment with 33 IT students included a complex hacker intrusion from the literature, illustrated either with MUCM or with alternative diagrams. The results suggest that participants using MUCM found significantly more mitigations than participants using regular misuse cases combined with system architecture diagrams.

With the continuously developing technology and growing complexity of software and systems, new demands and challenges appear for security, calling for new techniques and methods in addition to the already existing ones. The variety of initiatives and the variations in the characterizations makes it hard for users to select the most appropriate one for their needs. We propose a set of uniform characterizing dimensions with sub-categories for security requirements initiatives. The set is derived by analyzing classifications and comparison frameworks from review papers on modelling techniques for security requirements engineering. The dimensions can be used to guide context-dependent choices of initiatives and further research of their combination and integration.

Migration is a strategy used in preservation systems to make digital objects survive from the continuing evolution of technology. However, it is difficult for custodians to decide an objective migration solution, as heterogeneous data, many system requirements, and complex programs will affect the selection of the solutions. In this paper, we describe a multi-criteria decision making approach for the selection of migration solutions. The approach mainly uses the analytic hierarchy process (AHP) and the technique ordered preference by similarity to the ideal solution (TOPSIS). Finally, the viability of our approach is illustrated in a series of experiments to find the best migration solution for doc files.

Web service is considered as the de facto standard to develop various business applications, due to the independence from the associated network, architecture, platform, and organization. However, Web service is still dependent to various internal factors (e.g., the availability of trucks is a limiting factor for the shipping Web service) for satisfying the requirements of customers. The unavailability of such underlying factors affect the existing business agreements made with other service providers, which result in conflict between the service partners. Web service has to handle the conflicts automatically to retain its customers and to improve the business profit. This demo paper illustrates an argumentative agents based approach to handle conflicts between Web services on the fly, using a case study of purchase order scenario.

Requirements traceability is a challenge for modern software projects where task dependencies and technical expertise are spread across system developers, abstract model representations such as use cases, and a myriad of code artefacts. This paper presents an approach that monitors the navigation trails left by developers when building code artefacts to realise project use cases. These trails are analysed to generate a relevance ranking of entities that constitute a traceability link between uses cases and code artefacts and the developers responsible for them. Investigation in a software development scenario shows that a range of use case traceability questions can be answered through visualisations which present ordered relevance lists of the entities associated with use cases and by the use of trace graphs where the size of nodes show the importance, or 'information centrality', of system entities.

Do software engineering students need interdisciplinary skills? Do students learn different things from an interdisciplinary project work than from software development projects? How can a specific interdisciplinary project course be organized? This paper provides reflections about these questions, based on the experience gained through running a project-based interdisciplinary course thrice. This course is part of the master degree education at the NTNU, Trondheim, Norway. Students taking the course work in teams of five, often coming from study programs of quite different disciplines. There is no predefined project assignment, instead the teacher has only described an open-ended theme, within which different student teams then define their own project assignment. The paper provides some reflections and lessons learned that can be exploited for designing similar interdisciplinary team project courses in other universities.

A semiotic framework for evaluating the quality of conceptual models was proposed by (Lindland OI, Sindre G and Sølvberg A (1994) Understanding Quality in Conceptual Modelling, IEEE Software 11(2), 41–49) and has later been extended in several works. While the extensions have fixed some of the limitations of the initial framework, other limitations remain. In particular, the framework is too static in its view upon semantic quality, mainly considering models, not modelling activities, and comparing these models to a static domain rather than seeing the model as a facilitator for changing the domain. Also, the framework's definition of pragmatic quality is quite narrow, focusing on understanding, in line with the semiotics of Morris, while newer research in linguistics and semiotics has focused beyond mere understanding, on how the model is used and impact its interpreters. The need for a more dynamic view in the semiotic quality framework is particularly evident when considering process models, which themselves often prescribe or even enact actions in the problem domain, hence a change to the model may also change the problem domain directly. This paper discusses the quality framework in relation to active process models and suggests a revised framework based on this.

In the early phases of software development, such as problem analysis and requirements engineering, oral communication techniques play a central part. In many university education programmes, however, such oral communication techniques are often addressed only superficially. Part of the reason may be that they are costly to teach and assess in a more elaborate way, while written assignments are much more easily administered to a large class. In the graduate course TDT4250 Modeling of Information Systems at the NTNU, we attempted to address oral communication skills by delegating to the students to role-play both as analysts and customers, and to have them peer-assess each other's analyst performances. The learning intervention received encouraging ratings from the students in a post-task survey, indicating several interesting observations as well as ideas for further improvement

Many software engineering courses include all-term projects to convey principles relating to large-scale multi-person development. But even such projects will easily be too small and simple, unless a sufficient amount of study time is allocated to them. This time may be hard to find, especially in strictly programmed profession studies where a lot of general theory courses have to be taken. This paper reports on the experiences from a software engineering project where the solution to the above problem has been to have several courses share one project. This had some advantages. First of all, it allows time for a bigger and more complex project with reasonable sacrifices of "own time" in each of the participating courses. Equally important, it is possible to show connections between the courses. In spite of these advantages, there have also been problems with the project, still leaving room for improvement.

This paper conducts an empirical analysis of a semiotics-based quality framework for quality assuring information models. 192 participants were trained in the concepts of the quality framework, and used it to evaluate models represented in an extended Entity Relationship (ER) language. A randomised, double-blind design was used, in which each participant independently reviewed multiple models and each model was evaluated by multiple reviewers. A combination of quantitative and qualitative analysis techniques were used to evaluate the results, including reliability analysis, validity analysis, interaction analysis, influence analysis, defect pattern analysis and task accuracy analysis.. An analysis was also conducted of the framework's likelihood of adoption in practice. The study provides strong support for the validity of the framework and suggests that it is likely to be adopted in practice, but raises questions about its reliability and the ability of participants to use it to accurately identify defects. The research findings provide clear directions for improvement of the framework. The research methodology used provides a general approach to empirical validation of quality frameworks.

The course SIF8005 Object-Oriented Programming at the NTNU is in many respects taught in a quite traditional manner, with a well-known textbook, lectures in huge auditoria, and compulsory exercises that the students have to deliver to be allowed to sit a final written exam. The exercise part of the course is a mixture of individual exercises on a weekly basis, and a somewhat larger project to be done by groups of 4 students. This paper particularly discusses the project, which has become a huge success after some notable changes were made for the 2001 offering of the course. Then the assignment profile was changed from one of fixed requirements set by staff to an open assignment, where each group made a computer game according to their own preferences. This change eliminated many of the problems present in earlier offerings and increased student inspiration.

The traditional thrust of software quality assurance has been to use "brute force" testing at the end of development. However quality assurance in the early phases of development is an order of magnitude more cost effective than testing done at the end of development. This paper describes the introduction of quality reviews as a learning activity into an undergraduate requirements analysis course, and evaluates the effectiveness of this in improving learning. In the absence of any standard instrument for evaluating learning effectiveness, we developed a survey instrument called the Learning Effectiveness Survey. Using this instrument, learning effectiveness is evaluated in the context of the learning goals of the course (short term learning), and in the context of the broader educational programme and future working life (long term learning). It also provides feedback on the learning intervention and how it can be improved. While the intervention was moderately successful in its first application, a great deal was learned about how it could be improved next time.

The paper discusses the general purpose of analysis and evaluates OOA with respect to this, arguing that OOA does not deliver what it claims to do. The two major problems are that OOA often does not meet the full needs of the analysis phase, and that the transition to design is not always as easy as promised. The last point is illustrated by a solution to the OOPSLA conference registration problem. Due to the mentioned shortcomings, OOA/OOD was not found sufficient for forming the basis of a common development methodology for three Norwegian software producers in a technology transfer project with our university. The suggestion made is that OOA should become problem-oriented rather than target-oriented