What Happened?
Last month, FDA’s scientists published the toxicological reference value (TRV) for exposure to cadmium in the diet. This value is the amount of a chemical—in this case cadmium—a person can consume in their daily diet that would not be expected to cause adverse health effects and can be used for food safety decision-making. The TRV was based on a systematic review FDA scientists published last year. We will turn to the TRV itself in an upcoming blog but are focusing on the systematic review here.
In a May 2023 publication, experts in systematic reviews from the University of California San Francisco (UCSF) raised concerns about FDA’s “lack of compliance” from established procedures.
We discussed these concerns with FDA. They said:
- “The systematic review and the TRV” publication “have both undergone external peer review by a third-party and experts in the field.” The agency expects to publish the reviews on its website, and
- FDA “is working on developing a protocol for a systematic review of cardiovascular effects of cadmium exposure that will be published.”
Why It Matters
Systematic review is a method designed to collect and synthesize scientific evidence on specific questions to increase transparency and objectivity and provide conclusions that are more reliable and of higher confidence than traditional literature reviews. In particular, the National Academies of Sciences, Engineering, and Medicine have recommended the use of systematic reviews to establish values such as the TRV that may be used to inform regulatory decisions.
The National Toxicology Program (NTP) and others have developed specific methodologies to conduct systematic reviews. FDA’s authors said they followed NTP’s Office of Health Assessment and Translation (OHAT) handbook.
Unfortunately, FDA’s adherence to the methodology fell short on both transparency and objectivity grounds, undermining the credibility of its conclusions. Credibility is crucial because FDA’s authors stated that “this systematic review ultimately supports regulatory decisions and FDA initiatives, such as Closer to Zero, which identifies actions the agency will take to reduce exposures to contaminants like cadmium through foods.”
Our Take
We are pleased to see FDA conducting systematic reviews of chemical safety assessments and using more rigorous methods for evaluating scientific evidence with increased transparency and trust in the outcomes.
But while their efforts may be well intentioned, FDA’s methodology has several flaws as identified by UCSF experts—and the agency’s unexplained variations from the OHAT handbook sets a troublesome precedent. Although most systematic review methods provide some flexibility, the study authors made significant changes without proper justification. The UCSF experts explained that FDA “deviations from the method threaten the validity” of the outcome. They identified three major issues:
- Failure to publish a protocol before starting the systematic review. A publicly available protocol is fundamental to trusting the outcome of a systematic review. The approaches to evidence selection, evaluation, synthesis, and integration must be stated before reviewing the evidence so that the results of studies do not bias the evaluation.
- Inappropriate quality rating of animal and human studies to exclude outcomes from consideration. Contrary to OHAT recommendations, the FDA scientists conducted an initial rating of animal and human studies using their own expertise before conducting the risk of bias assessment that excluded outcomes from consideration to derive the TRV. The risk of bias assessment is a crucial component to rate the confidence in the evidence.
- Inappropriate removal of a large number of epidemiological studies for consideration. The authors set up their own criteria to assess the risk of bias of human studies. They excluded 23 of 30 studies in a manner that was inconsistent with their own criteria.
In a response to the critique of their study, FDA’s authors acknowledged that they had a predefined protocol and a plan they followed, but that the protocol and plan were not published. They acknowledged that they could have “better described” the rating and exclusion of studies.
FDA authors used a structured methodology designed to account for every decision made about the strength and confidence in the scientific evidence. Unfortunately, they had unexplained variations that diminished the credibility of the TRV for cadmium that would be protective of human health.
What’s Next?
We will continue to engage with the agency and evaluate claims that they have conducted systematic reviews with the goal of improving the credibility of their scientific conclusions.
Go Deeper
Read our blogs on systematic reviews.
Download a PDF copy of this blog post. (PDF, 372KB)