U.S. flag

An official website of the United States government

NIH Library National Institutes of Health Library

Risk of Bias

What It Is | How to Assess | Tools | Manuals & Guidance

What It Is

Assessing the risk of bias in evidence syntheses involves assessing the internal validity of individual studies included in your evidence synthesis.

The University of York's Centre for Reviews and Dissemination provide helpful definitions of key concepts:

  • "Internal validity is the extent to which an observed effect can be truly attributed to the intervention being evaluated, rather than to flaws in the design or conduct of the study. The assessment of internal validity of quantitative studies involves determining whether the methods used in the study can be trusted to provide a genuine, accurate account of the intervention” (CRD Manual).
  • "Bias refers to systematic deviations from the true underlying effect brought about by poor study design or poor conduct in the collection, analysis, interpretation, publication or review of data. Bias can obscure intervention effects, and differences in the risk of bias between studies can help explain differences in findings” (CRD Manual).

How to Assess

To assess the risk of bias, at least two review team members assess each included article independently using a previously selected tool. To resolve discrepancies, the team may use consensus discussion or a separate third reviewer. It is likely that questions and discussion will be necessary with both review team members.

As many tools are not in a format to easily or efficiently collect or review the risk of bias data, it is recommended that the team operationalize the tool into an online format (e.g., use Microsoft Forms). The NIH Librarian will not operationalize the tool for the review but can provide suggestions on software to use.

It is also highly recommended that the team pilot this step with all participating team members on a sample of at least 5 articles representing the various study designs and types of articles included in the review. If possible, those with the most subject matter expertise and familiarity with study design should complete this step. If that is not possible, start slowly by completing only a few articles initially and meeting frequently to discuss discrepancies and questions until team members are comfortable with the process.

Tools

A wide variety of tools (i.e., checklists) are available for conducting the risk of bias assessment. Select the tool that matches the study designs (e.g., cohort, case-control, RCT, etc.) or type (e.g., editorial) of your included articles.

If you have questions about tool selection, contact the NIH Library for assistance.

To find a collection of various tools, you can check out these two resources:

Additionally, these sources have checklists available for multiple study designs or article types:

  • JBI Critical Appraisal Tools (cross-sectional, case-control, case reports, case series, cohort, diagnostic, economic evaluation, prevalence, qualitative, quasi-experimental, RCTs, systematic reviews, text and opinion)
  • NHLBI Study Quality Assessment Tools (controlled interventions, systematic reviews, observational cohort, cross-sectional, case-control, before-after/pre-post with no control, case series)

Other tools frequently used to assess the risk of bias in evidence synthesis include:

  • AMSTAR 2: A MeaSurement Tool to Assess Systematic Reviews 2
  • Newcastle-Ottawa Scale (NOS):  assessing the quality of nonrandomised studies
  • ROBIS: Risk of Bias in Systematic Reviews
  • ROBINS-I: Risk Of Bias In Non-randomized Studies – of Interventions
  • ROBINS-E: Risk Of Bias of Non-randomized Studies – of Exposures
  • RoB 2.0: revised tool for Risk of Bias in Randomized Trials (parallel, cross-over, and others) from the Cochrane Collaboration
  • QUADAS-2: evaluate the risk of bias and applicability of primary diagnostic accuracy studies

Manuals & Guidance

For more details and guidance on assessing the risk of bias, please review the following resources. You can also ask a NIH Librarian for additional help.

Next > Data Collection

Eligibility Criteria | Screening | Using Covidence | Risk of Bias | Data Collection | Data Synthesis