Planning

Commissioning a Review

Organizations (e.g., software companies, government departments) may commission a systematic review if they have a well-defined question that can be answered by summarizing existing empirical evidence.

Software tool producers may want to know about recent developments that are related to the functions provided by their tool. Software quality managers and software process managers may want to have some reliable evidence about the impact of a new method/tool. Government organizations responsible for commissioning or purchasing software intensive systems may want information about the effectiveness and limitations of different software processes. Organizations may also be interested if there is no reliable empirical evidence about the benefits and limitation of a specific method/tool.

The process of commissioning involves:

• Justifying the need for the review.
• Specifying the research question(s).
• Preparing and publishing an invitation to tender.
• Assessing the responses to the invitation to tender.
• Awarding the contract.

Developing a review protocol

A review protocol specifies all the methods that will be used to conduct a specific systematic review. In some disciplines, review protocols must be formally registered and published prior to the start of the review, however, this is not common practice in Software Engineering.

A pre-defined protocol is an important element in terms of the ‘systematic’ aspect of a review, and is necessary to reduce the possibility of researcher bias. For example, without a protocol, it is possible that the selection of individual studies or the analysis may be driven by researcher expectations and prejudices.

The components of a protocol include descriptions of the plan for all the elements of the review plus some additional planning information, including:

• Background, including an introduction to the review topic and a justification for why a review is needed.
• Research Questions.
• Eligibility Criteria.
• Search Process.
• Selection Process.
• Data Definition and Data Extraction.
• Data Synthesis.
• Critical Appraisal of Primary Studies (Risk of Bias/Quality Assessment).
• Assessing the strength of evidence (Certainty Assessment).
• Project timetable and staffing information.
• The process used to evaluate the review protocol and the results of the review
• Dissemination and reporting strategy.

In addition, the protocol should identify any issues that require adapting the standard review process.

• If review authors have, themselves, contributed candidate empirical studies, the selection, risk of bias assessment and data extraction processes may need to be organized to ensure reviewers do not make critical assessments of their own empirical studies.
• If there are related systematic reviews and mapping studies, the protocol should specify how information from these studies will be used.
• The protocol should define how individual articles that report more than one empirical study are to be identified and processed. For systematic reviews, the usual procedure is to assess each different empirical study as a separate primary study. However, this is not usually required for mapping studies.

Constructing a protocol not only involves defining the methods that will be used to conduct the review, it also involves trying out different aspects of the methodology. In order to try out the processes, you need to identify a set of known primary studies. This can be based on your own knowledge, or preliminary Google Scholar searches for potential primary studies, or related secondary studies (which can provide multiple candidate primary studies). Known studies can be used to help validate the eligibility criteria, and to prototype the search process, the selection process, the data definition and data extraction and critical appraisal processes. In particular, the known studies can be used to:

• try out the eligibility criteria
• help identify appropriate key words for search strings
• assess the coverage achieved by different search strings and digital libraries
• identify an appropriate critical appraisal instrument based on the known primary study types(s) and to assess the agreement obtained by different members of the review team to assess its reliability (not usually required for mapping studies).
• try out the data collection process and assess its reliability.
• help to define the strength of evidence instrument and assess the process used to assess strength of evidence (not required for mapping studies).

Justifying the Need for the Review

This involves describing the review topic, and explaining why some form of secondary study addressing the topic could provide useful information to researchers or practitioners.

A systematic review or mapping study is an appropriate choice of methodology:

• If there is a large body of evidence that has not been organized, a mapping study is appropriate, for example a review of the development of mutation testing.
• If there is a reasonably large body of evidence but disagreements exist about an important issue, for example, to investigate whether software cost estimation models are better than expert opinion for estimating software development effort.
• If there is a new software engineering topic being discussed in the literature, qualitative systematic reviews are useful to investigate the main concepts and identify current terminology.
• If there is already an existing systematic review, but it needs updating to account for new empirical results.
• If you are starting a research project, a mapping study is a good starting point to organize and understand the literature in the topic area. It allows a researcher to identify potential research gaps and controversies in order to position their research with respect to existing research.

It should be clear who is likely to benefit from the results of the review, so that the dissemination process is geared to the appropriate audience. Members of the review team should know whether the review is intended for internal use, or for a specific client organization, or for external publication.

Specifying the research question(s)

Specifying the research questions is the most important part of any systematic review.

The review questions drive the entire systematic review methodology:

• The type of question affects which type of systematic review is appropriate, and the method that should be used for aggregating the evidence. A general question such as “What is the current status of Mutation Testing research?” would be addressed by a mapping study. A question such as “Is pair-programming more effective than code inspection at reducing code defects?” would be addressed by a quantitative systematic review. A question such is “What are the benefits and risks of adopting DevOps?” would be addressed by a qualitative systematic review.
• The eligibility criteria must be designed to exclude studies that do not address the research questions and include those studies that do.
• The search process must be designed to identify the primary studies that address the research questions.
• The data extraction process must extract the data items needed to answer the questions.
• The data analysis process must synthesize the data in such a way that the review questions can be answered.

The more specific the question, the easier it is find evidence that can answer the question. For example, if our question is  “Are cost estimates based on software estimation tools more accurate than cost estimates based on expert opinion?” we know that we need to look for software engineering studies that compare cost estimates obtained from tools with estimates obtained from experts[1]. This allows us to target our evidence better, for example: we can eliminate studies that compare the accuracy of estimates from different tools. However, if our question is “What is the best method for cost estimation?”, we may find many studies which compare many different types of cost estimation methods, each of which may use a different basis for comparison (e.g., most accurate, least risky, or most easy to use), so it may be difficult to know how to aggregate the evidence into a single answer.

However, if we ask a very specific question, it is extremely unlikely that we would find any existing evidence. For example a question such as “Is expert opinion better than a cost model to estimate the elapsed time a team of two developers using pair programming will take to design, code and test a major enhancement to a Java application?[2]” might define exactly what you would like to know, but it may be far too specific to find any relevant empirical evidence.

Generally, software engineering questions are rather broad, and need some additional information about context if the findings are to be relevant to software practitioners. In addition, because we seldom have multiple empirical studies addressing exactly the same research question, we often need to adopt qualitative rather than quantitative methods to aggregate results.

A specific problem that arises when formulating research questions is that software engineering terminology is not always well-defined, and is subject to change. For example, the term technical debt can be traced back to an article written in 1992, however, the concept was discussed from the 1980’s as one of the laws of system evolution, while in agile methods the term refactoring is used. Thus, using only the term technical debt to define the topic, risks missing relevant studies.

[1] We should point out that compared with other disciplines even this question is quite broad. We have not specified what type  of software product we want to estimate, e.g., a new application or an enhancement, nor what languages we are interested in e.g., C or Java or Cobol, nor whether we are concerned about point estimates or ranges, nor whether we are actually concerned with monetary cost, or staff effort or elapsed time.

[2] This is similar to the sort of detail that is used for health care questions!

Specifying Eligibility Criteria

Eligibility criteria are used to determine whether a candidate primary study can be included in the systematic review. Eligibility criteria are of two types:

1. Inclusion criteria, which specify properties that a primary study must possess.
2. Exclusion criteria, which specify properties that a primary study must not possess.

Eligibility criteria are derived from the SR topic and the SR research question(s). They should be defined well enough to ensure that different members of the review team can understand them and apply them consistently. They should be justified, in order to assure the SR reader that they are appropriate, given the goals of SR.

Particular criteria that need justification, are those based on:

• setting start and end dates for the search,
• allowing only reports of primary studies written in a specific language (which is usually English),
• restricting the search to a specific set of journals (usually high quality journals),
• eliminating grey literature – i.e., informally published articles such as theses and technical reports.

Evaluating the review protocol

The review protocol is an important document and should be properly reviewed by all members of the review team. It is important to detect possible flaws in the review plan before conducting the review itself.

Inviting independent experts to review a protocol can be extremely helpful, but usually they are extremely hard to find. For commissioned reviews there may be a requirement for external evaluation of the protocol, with allocation of funding to ensure that appropriate experts are willing to do the evaluation.

When reviewing a protocol, check that each of the required sections has been completed and assess issues such as whether:

• There is an appropriate justification for the review.
• The topic is appropriate for a systematic review.
• The systematic review process conforms to accepted guidelines and standards and any deviations are explained and justified.
• The study selection criteria are justified.
• The search strings are appropriately derived from the research questions.
• The identified data items will properly address the research question(s).
• The data analysis procedure is appropriate to answer the research question.