We present our work on testing access control of large national e-health Internet portal which has millions of monthly visits. Our aim is twofold: (1) to improve testing by applying systematic and rigorous (semi-formal) approach and (2) to obtain holistic view of portal’s complex access control structure. Applying more rigorous approach facilitates reducing ambiguity while holistic picture aids on easier and often also faster comprehension of complex access control structure. We use set-theoretic approach for specifying access control. Then, from access control’s abstract set notations we derive a visual version in form of the access control tree. Nodes of the tree represent attributes that influence access while edges are values of those attributes. The leaf of the tree represents a scope which is a grouping of individual services. Access control tree presented in this paper has 15 scopes (leaves) which results in 105 pairs of abstract test scenarios. Complete version of the tree has 66 scopes that result in over 2000 pairs of abstract test scenarios. Abstract test scenarios are implemented into over 600 concrete and automated test cases. Manual execution test of one concrete test takes about five minutes while automated execution of all tests takes about one hour (thus achieving over 40 times speedup). These automated test cases run as a part of our CI/CD pipeline. Access control tree can also be used as a collaboration or learning tool, to get quicker familiarity with the solution.

We present lightweight model-based testing (MBT) of privacy and authorization concepts of national portal for electronic health services in Norway (which has over a million of visits per month). We have developed test models for creating and updating privacy levels and authorization categories using finite state machine. Our models emphasize not only positive but also negative behavioral aspects of the system. Using edge and edge-pair coverage as an acceptance criteria we identify and systematically derive abstract tests (high level user scenario) from models. Abstract tests are further refined and transformed into concrete tests with detailed steps and data. Although derivation of abstract tests and their transformation into concrete ones are manual, execution of concrete tests and generation of test report are automated. In total, we extracted 85 abstract test cases which resulted in 80 concrete test cases with over 550 iterations. Automated execution of all tests takes about 1 hour, while manual execution of one takes about 5 minutes (over 40 times speedup). MBT contributed to shift the focus of our intellectual work effort into model design rather than test case design, thus making derivation of test scenarios systematic and straight forward. In addition, applying MBT augmented and extended our traditional quality assurance techniques by facilitating better comprehension of new privacy and authorization concepts. Graphical models served as a useful aid in learning these concepts for newcomers.

We present a process driven test automation solution which enables delegating (part of) automation tasks from test automation engineer (expensive resource) to test analyst (non-developer, less expensive). In our approach, a test automation engineer implements test steps (or actions) which are executed automatically. Such automated test steps represent user actions in the system under test and specified by a natural language which is understandable by a non-technical person. Then, a test analyst with a domain knowledge organizes automated steps combined with test input to create an automated test case. It should be emphasized that the test analyst does not need to possess programming skills to create, modify or execute automated test cases. We refine benchmark test automation architecture to be better suitable for an effective separation and sharing of responsibilities between the test automation engineer (with coding skills) and test analyst (with a domain knowledge). In addition, we propose a metric to empirically estimate cooperation between test automation engineer and test analyst's works. The proposed automation solution has been defined based on our experience in the development and maintenance of Helsenorg, the national electronic health services in Norway which has had over one million of visits per month past year, and we still use it to automate the execution of regression tests.