What is Important for Researchers?
In the context of software engineering research, the most important element of EBSE is the well-defined systematic review method. The systematic review process provides a method for rigorously and fairly analysing all of the available information about a given phenomenon. This provides a less biased and more complete perspective than would be obtained from a single study or an informal literature review. Taking a broad view of available evidence should make it possible to produce more reliable conclusions and to minimise the risk of bias.
Systematic reviews originated in clinical medicine, where it was recognised that the outcomes from individual experiments were not a safe or sufficient basis for decision-making. They were originally designed to take advantage of the rigorously defined experiments used in medical studies (referred to as randomised controlled trials, or RCTs). Subsequently their use has spread into other domains where such experimental rigour is either difficult or impossible to achieve, and where many different experimental forms may well be employed. Indeed, for that reason, many people prefer to use the term evidence-informed to indicate that the findings from a review may need to be adapted to a particular context.
Systematic reviews are now a standard research method widely used in research-based disciplines. As such they are an important method for all software engineering researchers.
What about Mapping Studies?
A mapping study (also termed a scoping review) provides a more ‘open’ form of review. This reviews a specific software engineering topic and classifies the primary research papers for that specific domain. The research questions for such a study are quite high-level, and can include issues such as which sub-topics have been investigated, what empirical forms have been used, and which sub-topics have been addressed by enough studies to merit a more detailed systematic review. The table below shows how a mapping study differs from a systematic review, with the main distinction being that where an SR seeks to aggregate the outcomes of primary studies, a mapping study aims only to classify literature and aggregate studies within the categories.
|Classification of available literature
|Identifying best practice, seeking consensus when empirical study finding disagree, thematic analysis and conceptual modelling of the literature.
|General — related to research trends: which researchers, how much activity, what type of studies, etc.
|Specific — related to the outcomes from empirical studies.
Of the form: “is technology / method A better or not than B”.Or of the form: “80% of high quality studies confirm the importance of factor X”.
|Defined by topic area
|Defined by research question
|Broad — all papers related to a topic area are included, but only classification data about those are collected
|Focused — only empirical papers related to a specific research question are included and detailed information about individual research outcomes is extracted from each paper
|Search strategy Requirements
|Less stringent if only research trends are of interest
|Extremely stringent — all relevant studies must be found
|Important to ensure that results are based upon best quality evidence
|Set of papers related to a topic area, categorised in a variety of dimensions and counts of the number of papers in various categories
|Answer to specific research question, possibly with qualifiers (e.g. results apply only to novices). Prioritised lists of important factors, qualitative models of the relationship among factors.
Are there Other Types of Systematic Review?
Two other types of systematic review that are of use in software engineering are:
- Tertiary studies. These are systematic reviews where the empirical studies are themselves secondary studies. See (Petersen, Vakkalanka & Kuzniarz, 2015), Guidelines for conducting systematic mapping studies in software engineering: An update. This form of review is often used to investigate how SE researchers are performing the systematic review process. However, they can also be used to compare and contrast different systematic reviews performed on the same topic. Tertiary studies are a useful source of information about secondary studies. See, for example, the paper by (Kamei et al., 2021), who analysed 446 systematic reviews published before 2019, in venues with a minimum h5-index (20 for conferences and 25 for journals). The list of papers, including abstracts, is available here. A sub-division of the tertiary review is the umbrella review, which “allows the findings of reviews relevant to a review question to be compared and contrasted”. An umbrella review can therefore be regarded as being a ‘systematic review of systematic reviews’. See (Aromataris et al., 2015)
- Rapid Reviews (RRs). Formally, a rapid review is a “form of knowledge synthesis that accelerates the process of conducting a traditional systematic review through streamlining or omitting various methods to produce evidence for stakeholders in a resource-efficient manner”. The first example of an RR in software engineering was reported by Cartaxo and his colleagues in 2018, see (Cartaxo, Pinto & Soares, 2018) The role of rapid reviews in supporting decision-making in software engineering practice. For a comparison of systematic reviews and rapid reviews, see (Caraxo, Pinto & Soares, 2020) Rapid Reviews in Software Engineering.
What Research Opportunities does this provide?
We encourage researchers to undertake both mapping studies and systematic literature reviews as part of the task of mapping out our empirical domain knowledge. Systematic reviews can help summarize areas where a large number of empirical studies have been performed, for example cost estimation and fault estimation. In addition, mapping studies, whether undertaken by others or not, can provide the basis for identifying where new primary studies are needed, perhaps because no-one has studied a particular aspect, or because a replication would provide valuable confidence about the findings.
Updated in February 2024