Introduction
Like general literature reviews, systematic reviews are reviews of the published scientific literature, bringing together evidence from various studies to answer a specific research question. The key difference between general literature reviews and systematic reviews is how they are conducted. The goal of systematic reviews is to minimize bias by following explicit pre-determined procedures for conducting the review and analysis. To this end, the Cochrane organization created a clear handbook and standards on how to properly conduct these systematic reviews, and provides support for researchers throughout the review process.
Systemic reviews are often used in biomedical or healthcare context. In this sense, most systematic reviews are concerned with the effects of a clinical intervention (i.e., drug treatment). Many authorities require systematic reviews when assessing the approval of new drugs or for insurance coverage, and have strict requirements in how these reviews are carried out and formatted – following Cochrane standards and PRISMA formatting. This standardized way of collecting and analyzing secondary data from published research studies is critical.
This summary will outline the steps of completing a Cochrane systematic review and the requirements of the PRISMA format, as well as provide links to helpful tools.
Conducting a Cochrane Systematic Review: First Steps
The first steps of a systematic review are to develop a specific research question and a descriptive title. The Cochrane Handbook emphasizes the importance of registering the systematic review with the Cochrane organization so that researchers will gain access to Cochrane’s support and tools. Cochrane also emphasizes the importance of building an author team covering different specialties.
The research question, or “Objective” as it is referred to by Cochrane, identifies the PICO elements (Population, Intervention, Comparators, and Outcome) of a health problem and is typically written in the following format:
‘To assess the effects of [intervention or comparison] for [health problem] in [types of people, disease or problem and setting if specified]’.
A well-formulated research question starts the review process on the right foot and will allow researchers to develop eligibility criteria for the later literature search and study selection steps. It will also help determine what types of comparisons are made in the systematic review and influences the analysis methodology.
Write the Review Protocol
The Review Protocol provides details on the full plan of the review and is what makes it “systematic”. It sets out the research question and scope of the review first and then outlines the FULL methodology. This includes exactly how the literature search process will be conducted, what criteria will be used to determine which studies to use in the review, what information will be extracted from these selected studies, and how to analyze the information. The methods proposed in the Review Protocol can change during the review process, but all these changes need to be documented, rationalized and reported.
Full protocols should be published separately from the final systematic review report. Past protocols can be accessed through the Cochrane Database of Systematic Reviews and through PROSPERO.
A properly drafted Review Protocol is crucial to the systematic review process. The FDA has a guidance document which provides some information on a systematic review marking scheme considered during regulatory decisions. The FDA evaluates such things as the pre-specification of objectives, eligibility criteria, choice of comparators, and outcomes; that trial outcomes were not used in determining if a study was selected to be part of the review; and appropriateness of statistical methods to combine trials and analyze sensitivity of data. All these attributes should be predetermined in the Review Protocol.
Determine Eligibility Criteria
Eligibility criteria are derived from the PICOs of the research question and help determine the terms used for conducting searches. These criteria allow authors to limit the review to the types of studies, participants, and interventions they are interested in. In this way, the concept of systematic review eligibility criteria is similar to the inclusion criteria of clinical trials.
Once studies have been amassed, the eligibility criteria also determine which studies are to be included in the review’s analysis. Eligibility criteria should be listed in order of importance, as this will speed up the study selection process later. If a single eligibility criterion is not met, the study is not eligible to be included in the review. Usually, the first few criteria will define the disease and conditions for establishing its presence in patients. These criteria are usually in the form of specific results from gold-standard diagnostic tests. The next criteria define the desired population group with a specific age range, sex or ethnicity. Other criteria usually relate to population characteristics that may influence an intervention’s effects such as disease severity levels.
Conduct the Literature Search
A Cochrane-style systematic review requires the literature search to be thorough, objective, and reproducible. To properly reduce bias, as many eligible studies as possible need to be collected from multiple sources. A reproducible search is carried out in a pre-specified and documented way, with the sources chosen beforehand and listed in the Review Protocol.
When preparing for a literature search, first identify the appropriate key words and terms (including spelling variations, synonyms, acronyms, etc.) These are determined by the PICOs of the research question and will specify the health conditions and population of interest, intervention-related terms, and terms related to the desired study design. If conducting searches through PubMed, these terms are often added through the MeSH tool.
The Cochrane Handbook says that validated filters may be used when searching for randomized studies in humans. Such filters have been developed by Cochrane for the MEDLINE and Embase databases and instructions on how to use them can be found in the online Technical Supplement. It may be necessary to specify a time period from which studies may be gathered but it is important to ensure that the systematic review is as up to date as possible.
The entire search process should be documented in enough detail so it can be reported in the review and be reproduced.
Examples of Places to Search for Studies:
CENTRAL (Cochrane Central Register of Controlled Trials) – a bibliographic database
MEDLINE – a bibliographic database; PubMed provides access for free
Embase – a bibliographic database; can be accessed through Elsevier, Ovid, ProQuest, or STN.
ClinicalTrials.gov – US clinical trails registry
International Clinical Trials Registry Platform (ICTRP) – WHO clinical trials registry
Others:
- Various databases focusing on specific subject areas: AMED (alternative therapies), CINAHL (nursing and allied health), and PsychINFO (psychology and psychiatry)
- Correspondence with study authors, especially to gain information on unpublished studies
Select Studies to Include in the Review
There are two overarching steps in the selection process. Relevant studies are to be included in the systematic review, but literature search results will be in the form of papers which document studies. The issue is that multiple papers may report the same study; and treating different reports of the same study as different studies will introduce bias into the systematic review. Therefore, the first step is to go through the collected documents and determine which are linked. Search results from different sources are merged and duplicate papers are removed. This can be done by looking at trial identification numbers, sponsor and author names, location and setting information, etc. Reference management software is helpful in this regard. Titles and abstracts can be used to remove obviously irrelevant papers, but the full text must be retrieved to determine exactly which papers refer to which studies.
The next step involves going through all the search results and determining which studies are to be included in the review based on the eligibility criteria. Cochrane requires at least two people to work independently and determine if each study meets the eligibility criteria. Titles and abstracts are often enough for determining eligibility, but full texts are often required as well.
When assessing the studies for eligibility, the criteria are listed and then when a single criterion is not met, the study is excluded, and the author can move on to assessing the next study without going through all criteria. However, this is not the end of the excluded study. A Cochrane systematic review should contain a table of “Characteristics of Excluded Studies”. This table will provide the specific reason why a study was excluded from the review. The table should also include studies that superficially appear eligible but do not meet more specific eligibility criteria, and any well-known relevant studies that are not eligible.
There are various software tools that can be used to help manage the process. The Cochrane organization prefers the use of the EPPI-Reviewer and Covidence tools.
Common Software Tools
EPPI-Reviewer – web-based software designed to support reference management, study screening, risk of bias assessments, data extraction and data synthesis steps of the systematic review process. It is free for Cochrane-registered reviews but otherwise requires a paid subscription with a free trial available.
Covidence – web-based platform that can provide support for title/abstract screening, full text review, risk of bias assessments, and data extraction. It is free for Cochrane review authors but otherwise requires a paid subscription with a free trial available.
Abstrackr – free web-based tool for the screening of records
DistillerSR – web-based application for screening and data extraction. It is not free but reduced pricing is available for Cochrane review authors.
Rayyan – free web-based application for citation screening and full text selection
RCT Classifier – A Cochrane-developed tool that can identify reports of randomized trials based on their titles and abstracts. It pulls from a dataset of records already screened by past Cochrane review authors.
Extract Data from Selected Studies
The next phase of the Cochrane systematic review process is extracting required information from the eligible studies in order to perform the later analyses. As with all other steps of conducting a Cochrane review, the extraction methods must be described in the Review Protocol. Data extraction is performed using structured collection forms (i.e. Google Forms) that relate back to the research question and allow for data required from each study to be retained in an organized way. The forms allow for data entry by multiple authors and provide a transparent historic record of the data used in the review.
To develop these forms, it is best to first develop outlines of the tables and figures that will be included in the final report. This will provide authors with an idea of what information will need to be extracted from the selected studies. Cochrane provides some helpful key points about developing questions for the forms: ask close-ended questions that define a list of permissible responses. This will also require the researcher to anticipate possible responses prior to developing the form. Questions should not be able to be left blank and instead should have “not applicable” or “not reported” options. If a judgement of the study is required, there should be a place to report the raw data from the study. A sample list of information that should be extracted from each study and included in the forms is as follows:
- Names of the people who filled out the form, and the relevant dates of completion
- Confirm that the study meets the eligibility criteria
- Recruitment and sampling procedures, enrollment start and end dates
- Information on randomization and masking, any controls for selection bias
- Statistical analysis information including the units, methods used, likelihoods of reporting bias
- Sources of funding and conflicts of interest of the authors
- Eligibility criteria used to recruit people into the study
- Geographic area where the study was conducted
- Characteristics of participants (i.e. numbers of, age, sex, socio-economic status, etc.)
- Intervention information including routes of delivery, dosing, timing, frequency, protocols, any descriptions of co-interventions
- Descriptions of any control groups
- Any specific procedures or equipment used in the interventions
- Measurement tools for determining clinical outcomes or endpoints
- Adverse events
- Statistical methods, subgroup analyses, etc.
- Key conclusions
Data extraction is a mostly manual process but electronic searches within text may be used to speed it up. However, this should not replace reading the report. At least two people must work independently to extract data from each study. Upon finishing, they verify and resolve discrepancies between them. Cochrane recommends the use of software to extract numerical data that is only presented in figures. There are many free options such as those listed below:
Software for Extracting Information from Figures
Determine Effect Measures
Effect measures are defined as statistical constructs that allows for comparing data from different intervention groups. These are presented with a confidence interval, P value and a standard error. In a systematic review, summary data from each of the selected studies is collected so that these effect measures can be determined and then compared between the studies in a consistent way.
It is important to choose the proper effect measures based on the type of extracted data. Therefore, the first step is to determine the data type. There are 5 general types: binary, continuous, ordinal, rates, and time-to-event (i.e. survival) data. Chapter 6 of the Cochrane Handbook outlines how to choose the proper effect measures based on these data types. A supplementary document, Statistical algorithms in Review Manager, also provides formulas to calculate these effect measures and their standard errors.
Assess Bias within Selected Studies
Cochrane-style systematic reviews are designed to bring the results of many different studies together to answer a research question in a transparent manner. One of the most important parts of this process is to consider the causes of bias and take steps to address it. Bias refers to systematic error where multiple replications of a study would reach the wrong conclusion and can make researchers under-estimate or over-estimate the effects of an intervention. There are two places in a systematic review where the effect of bias must be considered: within each individual study that was selected to be included in the review, and in the meta-analysis of the data that was extracted from these studies.
At least one specific result of each included study should be assessed for risk of bias. To assess this risk in randomized trials, Cochrane developed and recommends the use of the RoB 2 tool. At least two people should work independently to use RoB 2 for each included study. This tool will provide an output of the risk and the researchers will provide a judgement of each risk as “high”, “low”, or “some concern” and provide a justification for this judgement. All this information is to be reported and Cochrane recommends that this be in the form of tables and forest plots. In general, the studies considered higher risk should be weighted less in the meta-analysis of the systematic review.
Synthesize the Extracted Data
The extracted data must be brought together to draw final conclusions. Synthesis of the different study characteristics (PICOs) will be included in the form of tables showing the characteristics of the different studies.
The desired types of comparisons are detailed in the Review Protocol prior to starting the systematic review. During the synthesis step, the studies should be grouped together in terms of their PICOs with the extracted data shown in its own column. This allows for easy comparisons and for the researchers to see what data is available to be compared. At this point, the researcher may determine if the studies contribute enough data required for a meta-analysis, if it is necessary to include a meta-analysis in the systematic review, and if any modifications to the Review Protocol need to be made.
When making comparisons between studies, meta-analysis or not, researchers should identify key differences in the PICO characteristics of the included studies. Finally, it will be necessary to check that findings of effect and statistical significance presented in the review are consistent with those from the original studies.
Conduct a Meta-analysis
Depending on the type and quality of the extracted data, it may be necessary to conduct a meta-analysis to answer the systematic review’s research question. A meta-analysis takes data from multiple separate studies and combines them statistically. They normally result in a weighted average of the studies’ effect measurements. Conducting a meta-analysis is a complicated process but can provide added precision, answer questions that were not the focus of the original studies and settle conflicting claims between studies. However, if the meta-analysis does not properly consider study designs, biases, variation, and other factors they can provide very misleading results.
In general, conducting a meta-analysis is a two-phase process. In the first phase, a summary statistic is calculated for each selected study. This ensures the effects reported by the studies are communicated in the same way. For example, they may all be put in the form of a risk ratio or a difference between means. The form of these summary statistics will depend on the research question of the systematic review, the data researchers were able to extract, and should be pre-determined and communicated in the Review Protocol. In the second phase, these summary statistics are combined in a weighted average with the standard error determining its confidence interval. The summary statistics may all be weighted the same or the weighting may change based on perceived bias, missing data, or other factors. Cochrane recommends that meta-analyses be communicated in forest plots.
Summarize Findings and Grade Evidence
Cochrane-style systematic reviews use the GRADE (Grading of Recommendations Assessment Development and Evaluation) approach for assessing the quality of the evidence provided in the systematic review. GRADE assessments are based on the risk of bias, data inconsistency, indirectness, imprecision and publication bias. The quality of the evidence is assessed to be high, moderate, low, or very low. These assessments are completed with the GRADE Working Group’s GRADEpro GDT tool, and a handbook on using this tool is available to review authors.
This GRADE tool can also assist with development of “Summary of Findings” tables, which are a standard part of Cochrane-style systematic reviews and must be presented in acceptable formats. In these tables, researchers will present the findings of the systematic review and provide comparisons of different clinical interventions and their outcomes. The types of outcomes compared are determined by the research question and should have been pre-determined and described in the Review Protocol. Examples of these outcomes are measurements of morbidity, side effects, symptoms, quality of life, treatment-associated burdens, and costs. The “Summary of Findings” tables first provide brief descriptions of the subject population, intervention and comparison treatment. The tables then list the desirable and undesirable health outcomes and provide a measure for the risk of each of these intervention outcomes.
Report the Results of the Systematic Review: PRISMA
Once the systematic review is completed, the final step is reporting the results in a submission. In the past, many published systematic reviews were found to omit key information. To address this, PRISMA (Preferred Reporting for Items for Systematic Reviews and Meta-Analyses) was developed to set out guidelines for the proper reporting of systematic reviews. PRISMA provides a checklist for the format of all essential information to be included in systematic review reports. This is the format any submitted papers are recommended to be in for publication in a scientific journal or for submission to a regulatory agency. Below is a chart showing the various headings and sub-headings of a systematic review report written in the PRISMA format and descriptions of the required information for each:
| Title | The title should identify the report as a “systematic review” and identify the participants, interventions, comparators, outcomes, and study design. |
| Abstract or Summary | This should range from 700-1000 words in length and describe the systematic review in language comprehensible to general audience (avoid abbreviations, use generic or internationally recognized drug names, etc.) The abstract has separate headings for background, objectives, data sources, study selection, data extraction, data synthesis, and conclusions/limitations. |
| Introduction | Describe what is already known and the rationale for the review. State if the systematic review is new or an update |
| Objectives | State the research questions with reference to the participants, interventions, comparators, outcomes and study design. |
| METHODS | |
| Protocol and Registration | Provide registration information of the systematic review and where published review protocols may be found. |
| Eligibility Criteria | Provide the specific eligibility criteria used to evaluate studies and reports of studies for inclusion in the systematic review. Be sure to state studies that do not meet the criteria will be excluded |
| Information Sources | Describe all information sources (databases, etc.) that were used to gather reports in the search process. |
| Search | Detail the full electronic search strategy used for a least one major database used. Include the key words and search terms used. This should be able to be repeated the same way it was done. |
| Study Selection | Provide a statement on how studies were selected (ex. “Two reviewers independently preformed unblinded eligibility assessments.”) |
| Data Extraction | Detail the methods used to extract data from study reports. The form used to collect the data is to an included in an appendix. |
| Data Items | List and define all variables for data extracted from the studies. |
| Risk of Bias in Individual Studies | Provide the method for assessing bias for each included study or provide rationales for why this was not done. |
| Summary Measures | Specify what outcomes measures will be made. |
| Planned Methods of Analyses | Explain the statistical methods of handling and combining data. |
| Risk of Bias Across Studies | Describe how risk of bias could affect the cumulative evidence collected for the review. |
| Additional Analyses | Describe any additional statistical analyses conducted, such as sensitivity or subgroup analyses, meta-regression, etc.) |
| RESULTS | |
| Study Selection | Give total numbers of studies found, assessed for eligibility, and included in the review. Provide a table detailing the excluded studies and reasons why. Provide a flowchart for the study search and selection process. |
| Study Characteristics | Describe the characteristics (PICOs) of each included study. Often in the form of tables and a narrative summary. |
| Risk of Bias within Studies | Present data on the risk of bias for each included study. |
| Results of Individual Studies | Detail all outcomes considered by the systematic review. Present a simple summary of the data for each intervention group of the selected study with confidence intervals. Forest plots are ideal. Be sure to state when results weren’t reported directly. |
| Synthesis of Results | Provide narrative description of the main results of the systematic review, relating back to the study populations and study designs. |
| Risk of Bias across Studies | Present results of any assessment of risk of bias across studies. |
| Additional Analyses | Report any subgroup or sensitivity analyses that were conducted. |
| DISCUSSION | |
| Summary of Evidence | Summarize the main findings, including the strength of evidence for each main outcome, and comment on the applicability of the findings to patients. |
| Limitations | Discuss limitations of the systematic review. |
| Conclusions | Interpret the result in terms of other evidence, such as other similar systematic reviews. |
| Funding | Describe sources of funding. |
Conclusion
I described the general process of conducting a systematic review in the Cochrane-style and requirements of reporting it in PRISMA format. Further training in the Cochrane process can be found at http://training.cochrane.org. I included information on useful tools to be used when conducting a systematic review. Cochrane also provides a register of such tools with the Systematic Review Toolbox. Below, is a list of all the activities discussed above in the relative order of completion.
- Develop the research question(s) / objective(s) of the systematic review
- Build the author team and register the systematic review
- Write the Review Protocol
- Determine eligibility criteria
- Determine which places will be searched for studies
- Show search strategy and provide example of how to conduct it in specific databases
- Develop data collection forms to be used to gather data from relevant studies
- Determine and describe what statistical methods and analyses will be done
- Publish the Review Protocol
- Conduct literature searches of multiple databases and other sources using your search strategy
- Determine which literature results relate to which studies
- Use study eligibility criteria to assess what is relevant to your research question
- Use data collection forms to extract information from each of the selected studies
- Assess extracted data for risk of bias
- Determine if the data is sufficient for planned analysis
- Bring extracted data together, make comparisons, conduct the meta-analysis
- Summarize findings of the systematic review
- Determine the quality of the systematic review through a GRADE assessment
- Write up a report of the systematic review in PRISMA format
References
Higgins JPT, Lasserson T, Chandler J, Tovey D, Thomas J, Flemyng E, Churchill R. (2019). Methodological Expectations of Cochrane Intervention Reviews (MECIR): Standards for the conduct and reporting of new Cochrane Intervention Reviews, reporting of protocols and the planning, conduct and reporting of updates (updated October 2019). Cochrane. Available from https://community.cochrane.org/sites/default/files/uploads/inline-files/MECIR%20October%202019%20Final%20Online%20version_2.pdf
Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA. (2019). Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019). Cochrane. Available from www.training.cochrane.org/handbook.
Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6(7): e1000097. https://doi.org/10.1371/journal.pmed.1000097