What is a Systematic Literature Review?
A systematic literature review (SLR) is an independent academic method that aims to identify and evaluate all relevant literature on a topic in order to derive conclusions about the question under consideration. "Systematic reviews are undertaken to clarify the state of existing research and the implications that should be drawn from this." (Feak & Swales, 2009, p. 3) An SLR can demonstrate the current state of research on a topic, while identifying gaps and areas requiring further research with regard to a given research question. A formal methodological approach is pursued in order to reduce distortions caused by an overly restrictive selection of the available literature and to increase the reliability of the literature selected (Tranfield, Denyer & Smart, 2003). A special aspect in this regard is the fact that a research objective is defined for the search itself and the criteria for determining what is to be included and excluded are defined prior to conducting the search. The search is mainly performed in electronic literature databases (such as Business Source Complete or Web of Science), but also includes manual searches (reviews of reference lists in relevant sources) and the identification of literature not yet published in order to obtain a comprehensive overview of a research topic.
An SLR protocol documents all the information gathered and the steps taken as part of an SLR in order to make the selection process transparent and reproducible. The PRISMA flow-diagram support you in making the selection process visible.
In an ideal scenario, experts from the respective research discipline, as well as experts working in the relevant field and in libraries, should be involved in setting the search terms. As a rule, the literature is selected by two or more reviewers working independently of one another. Both measures serve the purpose of increasing the objectivity of the literature selection. An SLR must, then, be more than merely a summary of a topic (Briner & Denyer, 2012). As such, it also distinguishes itself from “ordinary” surveys of the available literature. The following table shows the differences between an SLR and an “ordinary” literature review.
Differences to "common" literature reviews
| Characteristic | SLR | common literature overvie |
|---|---|---|
| Independent research method | yes | no |
| Explicit formulation of the search objectives | yes | no |
| Identification of all publications on a topic | yes | no |
| Defined criteria for inclusion and exclusion of publications | yes | no |
| Description of search procedure | yes | no |
| Literature selection and information extraction by several persons | yes | no |
| Transparent quality evaluation of publications | yes | no |
What are the objectives of SLRs?
- Avoidance of research redundancies despite a growing amount of publications
- Identification of research areas, gaps and methods
- Input for evidence-based management, which allows to base management decisions on scientific methods and findings
- Identification of links between different areas of researc
Process steps of an SLR
A SLR has several process steps which are defined differently in the literature (Fink 2014, p. 4; Guba 2008, Transfield et al. 2003). We distinguish the following steps which are adapted to the economics and management research area:
1. Defining research questions
Briner & Denyer (2009, p. 347ff.) have developed the CIMO scheme to establish clearly formulated and answerable research questions in the field of economic sciences:
C – CONTEXT: Which individuals, relationships, institutional frameworks and systems are being investigated?
I – Intervention: The effects of which event, action or activity are being investigated?
M – Mechanisms: Which mechanisms can explain the relationship between interventions and results? Under what conditions do these mechanisms take effect?
O – Outcomes: What are the effects of the intervention? How are the results measured? What are intended and unintended effects?
The objective of the systematic literature review is used to formulate research questions such as “How can a project team be led effectively?”. Since there are numerous interpretations and constructs for “effective”, “leadership” and “project team”, these terms must be particularized.
With the aid of the scheme, the following concrete research questions can be derived with regard to this example:
Under what conditions (C) does leadership style (I) influence the performance of project teams (O)?
Which constructs have an effect upon the influence of leadership style (I) on a project team’s performance (O)?
Research questions do not necessarily need to follow the CIMO scheme, but they should:
- ... be formulated in a clear, focused and comprehensible manner and be answerable;
- ... have been determined prior to carrying out the SLR;
- ... consist of general and specific questions.
As early as this stage, the criteria for inclusion and exclusion are also defined. The selection of the criteria must be well-grounded. This may include conceptual factors such as a geographical or temporal restrictions, congruent definitions of constructs, as well as quality criteria (journal impact factor > x).
2. Selecting databases and other research sources
The selection of sources must be described and explained in detail. The aim is to find a balance between the relevance of the sources (content-related fit) and the scope of the sources.
In the field of economic sciences, there are a number of literature databases that can be searched as part of an SLR. Some examples in this regard are:
Our video "Selecting the right databases" explains how to find relevant databases for your topic.
Literature databases are an important source of research for SLRs, as they can minimize distortions caused by an individual literature selection (selection bias), while offering advantages for a systematic search due to their data structure. The aim is to find all database entries on a topic and thus keep the retrieval bias low (tutorial on retrieval bias). Besides articles from scientific journals, it is important to inlcude working papers, conference proceedings, etc to reduce the publication bias (tutorial on publication bias).
Our online self-study course "Searching economic databases" explains step 2 und 3.
The selection of sources must be described and explained in detail. The aim is to find a balance between the relevance of the sources (content-related fit) and the scope of the sources.
3. Defining search terms
Once the literature databases and other research sources have been selected, search terms are defined. For this purpose, the research topic/questions is/are divided into blocks of terms of equal ranking. This approach is called the block-building method (Guba 2008, p. 63). The so-called document-term matrix, which lists topic blocks and search terms according to a scheme, is helpful in this regard. The aim is to identify as many different synonyms as possible for the partial terms. A precisely formulated research question facilitates the identification of relevant search terms. In addition, keywords from particularly relevant articles support the formulation of search terms.
A document-term matrix for the topic “The influence of management style on the performance of project teams” is shown in this example.
Identification of headwords and keywords
When setting search terms, a distinction must be made between subject headings and keywords, both of which are described below:
Keywords
- appear in the title, abstract and/or text
- sometimes specified by the author, but in most cases automatically generated
- non-standardized
- different spellings and forms (singular/plural) must be searched separately
Subject headings
- describe the content
- are generated by an editorial team
- are listed in a standardized list (thesaurus)
- may comprise various keywords
- include different spellings
- database-specific
Subject headings are a standardized list of words that are generated by the specialists in charge of some databases. This so-called index of subject headings (thesaurus) helps searchers find relevant articles, since the headwords indicate the content of a publication. By contrast, an ordinary keyword search does not necessarily result in a content-related fit, since the database also displays articles in which, for example, a word appears once in the abstract, even though the article’s content does not cover the topic.
Nevertheless, searches using both headwords and keywords should be conducted, since some articles may not yet have been assigned headwords, or errors may have occurred during the assignment of headwords.
To add headwords to your search in the Business Source Complete database, please select the Thesaurus tab at the top. Here you can find headwords in a new search field and integrate them into your search query. In the search history, headwords are marked with the addition DE (descriptor).
The EconBiz database of the German National Library of Economics (ZBW – Leibniz Information Centre for Economics), which also contains German-language literature, has created its own index of subject headings with the STW Thesaurus for Economics. Headwords are integrated into the search by being used in the search query.
Since the indexes of subject headings divide terms into synonyms, generic terms and sub-aspects, they facilitate the creation of a document-term matrix. For this purpose it is advisable to specify in the document-term matrix the origin of the search terms (STW Thesaurus for Economics, Business Source Complete, etc.).
Searching in literature databases
Once the document-term matrix has been defined, the search in literature databases begins. It is recommended to enter each word of the document-term matrix individually into the database in order to obtain a good overview of the number of hits per word. Finally, all the words contained in a block of terms are linked with the Boolean operator OR and thereby a union of all the words is formed. The latter are then linked with each other using the Boolean operator AND. In doing so, each block should be added individually in order to see to what degree the number of hits decreases.
Since the search query must be set up separately for each database, tools such as LitSonar have been developed to enable a systematic search across different databases. LitSonar was created by Professor Dr. Ali Sunyaev (Institute of Applied Informatics and Formal Description Methods – AIFB) at the Karlsruhe Institute of Technology.
Advanced search
Certain database-specific commands can be used to refine a search, for example, by taking variable word endings into account (*) or specifying the distance between two words, etc. Our overview shows the most important search commands for our top databases.
Additional searches in sources other than literature databases
In addition to literature databases, other sources should also be searched. Fink (2014, p. 27) lists the following reasons for this:
- the topic is new and not yet included in indexes of subject headings;
- search terms are not used congruently in articles because uniform definitions do not exist;
- some studies are still in the process of being published, or have been completed, but not published.
Therefore, further search strategies are manual search, bibliographic analysis, personal contacts and academic networks (Briner & Denyer, p. 349). Manual search means that you go through the source information of relevant articles and supplement your hit list accordingly. In addition, you should conduct a targeted search for so-called gray literature, that is, literature not distributed via the book trade, such as working papers from specialist areas and conference reports. By including different types of publications, the so-called publication bias (DBWM video “Understanding publication bias”) – that is, distortions due to exclusive use of articles from peer-reviewed journals – should be kept to a minimum.
The PRESS-Checklist can support you to check the correctness of your search terms.
4. Merging hits from different databases
In principle, large amounts of data can be easily collected, structured and sorted with data processing programs such as Excel. Another option is to use literature management programs such as EndNote, Citavi or Zotero. The Saxon State and University Library Dresden (SLUB Dresden) provides an overview of current literature management programs . Software for qualitative data analysis such as NVivo is equally suited for data processing. A comprehensive overview of the features of different tools that support the SLR process can be found in Bandara et al. (2015).
Our online-self study course "Managing literature with Citavi" shows you how to use the reference management software Citavi.
When conducting an SLR, you should specify for each hit the database from which it originates and the date on which the query was made. In addition, you should always indicate how many hits you have identified in the various databases or, for example, by manual search.
Exporting data from literature databases
Exporting from literature databases is very easy. In Business Source Complete , you must first click on the “Share” button in the hit list, then “Email a link to download exported results” at the very bottom and then select the appropriate format for the respective literature program.
In the Web of Science database, you must select “Export” and select the relevant format. Tip: You can adjust the extracted data fields. Since for example the abstract is not automatically exported, decide which data fields are of interest for you.
Exporting data from the literature database EconBiz is somewhat more complex. Here you must first create a marked list and then select each hit individually and add it to the marked list. Afterwards, articles on the list can be exported.
After merging all hits from the various databases, duplicate entries (duplicates) are deleted.
5. Applying inclusion and exclusion criteria
All publications are evaluated in the literature management program applying the previously defined criteria for inclusion and exclusion. Only those sources that survive this selection process will subsequently be analyzed. The review process and inclusion criteria should be tested with a small sample and adjustments made if necessary before applying it to all articles. In the ideal case, even this selection would be carried out by more than one person, with each working independently of one another. It needs to be made clear how discrepancies between reviewers are dealt with.
The review of the criteria for inclusion and exclusion is primarily based on the title, abstract and subject headings in the databases, as well as on the keywords provided by the authors of a publication in the first step. In a second step the whole article / source will be read.
Within the Citavi literature-management program, you can supplement title data by adding your own fields. In this regard, the criteria for inclusion can be listed individually and marked with 0 in the free text field for being “not fulfilled” and with 1 for being “fulfilled”. In the table view of all titles, you can use the column function to select which columns should be displayed. Here you can include the criteria for inclusion. By exporting the title list to Excel, it is easy to calculate how many titles remain when applying the criteria for inclusion and exclusion.
In addition to the common literature management tools, you can also use software tools that have been developed to support SLRs. The central library of the university in Zurich has published an overview and evaluation of different tools based on a survey among researchers. --> View SLR tools
The selection process needs to be made transparent. The PRISMA flow diagram supports the visualization of the number of included / excluded studies.
Forward and backward search
Should it become apparent that the number of sources found is relatively small, or if you wish to proceed with particular thoroughness, a forward-and-backward search based on the sources found is recommendable (Webster & Watson 2002, p. xvi). A backward search means going through the bibliographies of the sources found. A forward search, by contrast, identifies articles that have cited the relevant publications. The Web of Science and Scopus databases can be used to perform citation analyses.
6. Perform the review
As the next step, the remaining titles are analyzed as to their content by reading them several times in full. Information is extracted according to defined criteria and the quality of the publications is evaluated. If the data extraction is carried out by more than one person, a training ensures that there will be no differences between the reviewers.
Depending on the research questions there exist diffent methods for data abstraction (content analysis, concept matrix etc.). A so-called concept matrix can be used to structure the content of information (Webster & Watson 2002, p. xvii). The image to the right gives an example of a concept matrix according to Becker (2014).
Particularly in the field of economic sciences, the evaluation of a study’s quality cannot be performed according to a generally valid scheme, such as those existing in the field of medicine, for instance. Quality assessment therefore depends largely on the research questions.
Based on the findings of individual studies, a meta-level is then applied to try to understand what similarities and differences exist between the publications, what research gaps exist, etc. This may also result in the development of a theoretical model or reference framework.
Example concept matrix (Becker 2013) on the topic Business Process Management
| Article | Pattern | Configuration | Similarities |
|---|---|---|---|
| Thom (2008) | x | ||
| Yang (2009) | x | x | |
| Rosa (2009) | x | x |
7. Synthesizing results
Once the review has been conducted, the results must be compiled and, on the basis of these, conclusions derived with regard to the research question (Fink 2014, p. 199ff.). This includes, for example, the following aspects:
- historical development of topics (histogram, time series: when, and how frequently, did publications on the research topic appear?);
- overview of journals, authors or specialist disciplines dealing with the topic;
- comparison of applied statistical methods;
- topics covered by research;
- identifying research gaps;
- developing a reference framework;
- developing constructs;
- performing a meta-analysis: comparison of the correlations of the results of different empirical studies (see for example Fink 2014, p. 203 on conducting meta-analyses)
Publications about the method
Bandara, W., Furtmueller, E., Miskon, S., Gorbacheva, E., & Beekhuyzen, J. (2015). Achieving Rigor in Literature Reviews: Insights from Qualitative Data Analysis and Tool-Support. Communications of the Association for Information Systems. 34(8), 154-204.
Booth, A., Papaioannou, D., and Sutton, A. (2012) Systematic approaches to a successful literature review. London: Sage.
Briner, R. B., & Denyer, D. (2012). Systematic Review and Evidence Synthesis as a Practice and Scholarship Tool. In Rousseau, D. M. (Hrsg.), The Oxford Handbook of Evidenence Based Management. (S. 112-129). Oxford: Oxford University Press.
Durach, C. F., Wieland, A., & Machuca, Jose A. D. (2015). Antecedents and dimensions of supply chain robustness: a systematic literature review. International Journal of Physical Distribution & Logistic Management, 46 (1/2), 118-137. doi: https://doi.org/10.1108/IJPDLM-05-2013-0133
Feak, C. B., & Swales, J. M. (2009). Telling a Research Story: Writing a Literature Review. English in Today's Research World 2. Ann Arbor: University of Michigan Press. doi: 10.3998/mpub.309338
Fink, A. (2014). Conducting Research Literature Reviews: From the Internet to Paper (4. Aufl.). Los Angeles, London, New Delhi, Singapore, Washington DC: Sage Publication.
Fisch, C., & Block, J. (2018). Six tips for your (systematic) literature review in business and management research. Management Review Quarterly, 68, 103–106 (2018). doi.org/10.1007/s11301-018-0142-x
Guba, B. (2008). Systematische Literaturrecherche. Wiener Medizinische Wochenschrift, 158 (1-2), S. 62-69. doi: doi.org/10.1007/s10354-007-0500-0 Hart, C. Doing a literature review: releasing the social science research imagination. London: Sage.
Jesson, J. K., Metheson, L. & Lacey, F. (2011). Doing your Literature Review - traditional and Systematic Techniques. Los Angeles, London, New Delhi, Singapore, Washington DC: Sage Publication.
Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. doi: 10.1136/bmj.n71.
Petticrew, M. and Roberts, H. (2006). Systematic Reviews in the Social Sciences: A Practical Guide. Oxford:Blackwell. Ridley, D. (2012). The literature review: A step-by-step guide. 2nd edn. London: Sage.
Chang, W. and Taylor, S.A. (2016), The Effectiveness of Customer Participation in New Product Development: A Meta-Analysis, Journal of Marketing, American Marketing Association, Los Angeles, CA, Vol. 80 No. 1, pp. 47–64.
Tranfield, D., Denyer, D. & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British Journal of Management, 14 (3), S. 207-222. doi: https://doi.org/10.1111/1467-8551.00375
Webster, J., & Watson, R. T. (2002). Analyzing the Past to Prepare for the Future: Writing a Literature Review. Management Information Systems Quarterly, 26(2), xiii-xxiii. http://www.jstor.org/stable/4132319
Examples
What is particularly good about this example is that search terms were defined by a number of experts and the review was conducted by three researchers working independently of one another. Furthermore, the search terms used have been very well extracted and the procedure of the literature selection very well described.
On the downside, the restriction to English-language literature brings the language bias into play, even though the authors consider it to be insignificant for the subject area.
- Only very specific keywords used
- No precise information on how the review process was carried out (who reviewed articles?)
- Only journals with impact factor (publication bias)
- Research questions are explicitly presented
- Search string very detailed
- Exact description of the review process
- 2 persons conducted the review independently of each other
