Coordinating Requirements Engineering and Software Testing

If everything you try works, you aren't trying hard enough.
Gordon Moore - Co-founder, Intel Corporation


The development of large, software-intensive systems is a complex undertaking that is generally tackled by a divide and conquer strategy. Organizations face thereby the challenge of coordinating the resources which enable the individual aspects of software development, commonly solved by adopting a particular process model. The alignment between requirements engineering (RE) and software testing (ST) activities is of particular interest as those two aspects are intrinsically connected: requirements are an expression of user / customer needs while testing increases the likelihood that those needs are actually satisfied.

The work in this thesis is driven by empirical problem identification, analysis and solution development towards two main objectives. The first is to develop an understanding of RE and ST alignment challenges and characteristics. Building this foundation is a necessary step that facilitates the second objective, the development of solutions relevant and scalable to industry practice that improve REST alignment.

The research methods employed to work towards these objectives are primarily empirical. Case study research is used to elicit data from practitioners while technical action research and field experiments are conducted to validate the developed solutions in practice.

This thesis contains four main contributions: (1) An in-depth study on REST alignment challenges and practices encountered in industry. (2) A conceptual framework in the form of a taxonomy providing constructs that further our understanding of REST alignment. The taxonomy is operationalized in an assessment framework, REST-bench (3), that was designed to be lightweight and can be applied as a postmortem in closing development projects. (4) An extensive investigation into the potential of information retrieval techniques to improve test coverage, a common REST alignment challenge, resulting in a solution prototype, risk-based testing supported by topic models (RiTTM).

REST-bench has been validated in five cases and has shown to be efficient and effective in identifying improvement opportunities in the coordination of RE and ST. Most of the concepts operationalized from the REST taxonomy were found to be useful, validating the conceptual framework. RiTTM, on the other hand, was validated in a single case experiment where it has shown great potential, in particular by identifying test cases that were originally overlooked by expert test engineers, improving effectively test coverage.

Key Information

PhD-Thesis, PDF.

Defense date and location: June 9, 2015, 1:15pm. Room J1650.

Venue: Campus Gräsvik, J Building, Blekinge Institute of Technology, Karlskrona, Sweden

Opponent: Associate Professor Martin Robillard (McGill University, Canada)

Members of the grading committee:

Substitute: Professor Lars Lundberg (Blekinge Institute of Technology, Sweden)

Chairperson: Professor Tony Gorschek (Blekinge Institute of Technology, Sweden)

List of papers included in the thesis

Chapter 2: Challenges and Practices in Aligning Requirements with Verification and Validation: A Case Study of Six Companies

E. Bjarnason, P. Runeson, M. Borg, M. Unterkalmsteiner, E. Engström, B. Regnell, G. Sabaliauskaite, A. Loconsole, T. Gorschek, and R. Feldt

Empirical Software Engineering, vol. 19, no. 6, pp. 1809–1855, 2014.

Abstract | Download

Weak alignment of requirements engineering (RE) with verification and validation (VV) may lead to problems in delivering the required products in time with the right quality. For example, weak communication of requirements changes to testers may result in lack of verification of new requirements and incorrect verification of old invalid requirements, leading to software quality problems, wasted effort and delays. However, despite the serious implications of weak alignment research and practice both tend to focus on one or the other of RE or VV rather than on the alignment of the two. We have performed a multi-unit case study to gain insight into issues around aligning RE and VV by interviewing 30 practitioners from 6 software developing companies, involving 10 researchers in a flexible research process for case studies. The results describe current industry challenges and practices in aligning RE with VV, ranging from quality of the individual RE and VV activities, through tracing and tools, to change control and sharing a common understanding at strategy, goal and design level. The study identified that human aspects are central, i.e. cooperation and communication, and that requirements engineering practices are a critical basis for alignment. Further, the size of an organisation and its motivation for applying alignment practices, e.g. external enforcement of traceability, are variation factors that play a key role in achieving alignment. Our results provide a strategic roadmap for practitioners improvement work to address alignment challenges. Furthermore, the study provides a foundation for continued research to improve the alignment of RE with VV.

Chapter 3: A Taxonomy for Requirements Engineering and Software Test Alignment

M. Unterkalmsteiner, R. Feldt, and T. Gorschek

ACM Transactions on Software Engineering and Methodology, vol. 23, no. 2, 2014.

Abstract | Download

Requirements Engineering and Software Testing are mature areas and have seen a lot of research. Nevertheless, their interactions have been sparsely explored beyond the concept of traceability. To fill this gap, we propose a definition of requirements engineering and software test (REST) alignment, a taxonomy that characterizes the methods linking the respective areas, and a process to assess alignment. The taxonomy can support researchers to identify new opportunities for investigation, as well as practitioners to compare alignment methods and evaluate alignment, or lack thereof. We constructed the REST taxonomy by analyzing alignment methods published in literature, iteratively validating the emerging dimensions. The resulting concept of an information dyad characterizes the exchange of information required for any alignment to take place. We demonstrate use of the taxonomy by applying it on five in-depth cases and illustrate angles of analysis on a set of thirteen alignment methods. In addition, we developed an assessment framework (REST-bench), applied it in an industrial assessment, and showed that it, with a low effort, can identify opportunities to improve REST alignment. Although we expect that the taxonomy can be further refined, we believe that the information dyad is a valid and useful construct to understand alignment.

Chapter 4: Assessing Requirements Engineering and Software Test Alignment with REST-bench - Five Case Studies

M. Unterkalmsteiner, T. Gorschek, R. Feldt, and E. Klotins

Journal of Systems and Software, Under Revision

Abstract | Download

The development of large, software-intensive systems is a complex undertaking that we generally tackle by a divide and conquer strategy. Companies thereby face the challenge of coordinating individual aspects of software development, in particular between requirements engineering (RE) and software testing (ST). A lack of REST alignment can not only lead to wasted effort but also to defective software. However, before a company can improve the mechanisms of coordination they need to be understood first. With REST-bench we aim at providing an assessment tool that illustrates the coordination in software development projects and identify concrete improvement opportunities. We have developed REST-bench on the sound fundamentals of a taxonomy on REST alignment methods and validated the method in a series of case studies. Following the principles of technical action research, we collaborated with five companies, applying REST-bench and iteratively improving the method based on the lessons we learned. We applied REST-bench both in Agile and Tayloristic environments, in projects lasting from weeks to years, and staffed as large as 1000 employees. The improvement opportunities we identified and the feedback we received indicate that the assessment was effective and efficient. Furthermore, participants confirmed that their understanding on the coordination between RE and ST improved.

Chapter 5: An Industrial Case Study on the Use of Test Cases as Requirements

E. Bjarnason, M. Unterkalmsteiner, E. Engström, and M. Borg

Accepted at the 16th International Conference on Agile Software Development, XP 2015.

Abstract | Download

It is a conundrum that agile projects can succeed ‘without requirements’ when weak requirements engineering (RE) is a known common cause for project failures. While Agile development projects often manage well without extensive requirements documentation, test cases are commonly used as requirements. We have investigated this agile practice at three companies in order to understand how test cases can fill the role of requirements. We performed a case study based on interview data from a previous study. The findings include a range of benefits and challenges in using test cases for eliciting, validating, verifying, tracing and managing requirements. In addition, we identified three scenarios for applying the practice, namely as a mature practice, as a de facto practice and as part of an agile transition. The findings provide insights into how the role of requirements may be met within an agile development project including challenges to consider.

Chapter 6: Large-scale Information Retrieval in Software Engineering - An Experience Report from Industrial Application

M. Unterkalmsteiner, T. Gorschek, R. Feldt, and N. Lavesson

Empirical Software Engineering, Under Revision [accepted for publication October 20, 2015]

Abstract | Download

Software Engineering activities are information intensive. Research proposes Information Retrieval (IR) techniques to support engineers in their daily tasks, such as establishing and maintaining traceability links, fault identification, and software maintenance. We describe an engineering task, test case selection, and illustrate our problem analysis and solution discovery process. The objective of the study is to gain an understanding of to what extent IR techniques (one potential solution) can be applied to test case selection and provide decision support in a large-scale, industrial setting. We analyze, in the context of the studied company, how test case selection is performed and design a series of experiments evaluating the performance of different IR techniques. Each experiment provides lessons learned from implementation, execution, and results, feeding to its successor. The three experiments led to the following observations: 1) there is a lack of research on parameter optimization of IR techniques for software engineering problems; 2) scaling IR techniques to industry data is challenging and for certain techniques still unsolved; 3) the IR context poses constraints on the empirical evaluation of IR techniques, requiring more research on developing valid statistical approaches. We believe that our experiences in conducting a series of IR experiments with industry grade data are valuable for peer researchers so that they can avoid the pitfalls that we have encountered. Furthermore, we identified challenges that need to be addressed in order to bridge the gap between laboratory IR experiments and real applications of IR in the industry.

Chapter 7: Supporting Experts in Test Case Selection with Topic Models

M. Unterkalmsteiner, R. Feldt, T. Gorschek

Under submission [under revision since July 1, 2015] to Software: Testing, Verification and Reliability

Abstract | Download

Test case selection is one possibility to focus the regression testing effort to the part of the system under test which is likely to exhibit faults. While there exist many test case selection techniques, the majority relies upon structural knowledge of the system or detailed design documentation. Our goal is to support test engineers in their selection task by combining their domain expertise with the ability to analyze large amounts of textual data originating from system test cases. We create topic models and exploit their probabilistic nature to reorganize an existing test case categorization, facilitating the discovery of relevant but unexpected test cases from a pool of existing test cases. We developed RiTTM, risk-based testing supported by topic models, in collaboration with our industry partner where we also evaluated the approach in a case study. RiTTM was applied by a non-domain expert on a database containing 1415 test cases, and identifying 29 relevant test cases that were not chosen originally by a domain expert test engineer. Seventeen of these test cases were serendipitous, that is, they were found to be unexpected.