Ryan O’Connell is a High Meadows Fellow; Jennifer McPartland, Ph.D., is a Senior Scientist.
Last night, Environmental Defense Fund (EDF) submitted critical comments on EPA’s Office of Pollution Prevention and Toxics’ (OPPT) “systematic review” document that OPPT is using to evaluate chemicals’ risks under the Toxic Substances Control Act (TSCA).
Systematic review, a hallmark of the clinical sciences, employs structured approaches to identifying, evaluating, and integrating evidence in a manner that promotes scientific rigor, consistency, transparency, objectivity, and reduction of bias.
Unfortunately, OPPT’s systematic review document deviates dramatically from the best practices in systematic review—practices developed over decades based on empirical evidence and experience in application. OPPT’s approach also significantly diverges from recent recommendations of the National Academy of Sciences (see here and here).
The method that OPPT uses to judge individual study quality – a numerical scoring strategy – is arbitrary and vague. It is also rejected by all leading systematic review frameworks. OPPT’s scoring strategy is likely to lead EPA to exclude studies in a manner inconsistent with TSCA’s requirements to use the best available science and weight of the scientific evidence.
Through this document, OPPT has introduced yet another way to dismiss important research that extends even beyond its proposed Censored Science rule.
Among EDF’s concerns detailed in our comments are the following:
- OPPT’s document is not consistent with TSCA requirements:
- OPPT’s scoring approach will result in a failure of EPA to use the best available science.
- EPA cannot exclude any reasonably available information from consideration in chemical risk evaluations.
- OPPT’s document does not constitute systematic review as established by leading experts and scientific institutions:
- EPA has failed to develop individual protocols for the first 10 chemicals undergoing risk evaluation.
- EPA has failed to describe its approach to data integration for the first 10 chemicals.
- OPPT should not use a scoring methodology to evaluate individual studies.
- OPPT’s approach to weighting study criteria is inconsistent with best practices in systematic review, and is unscientific and flatly arbitrary.
- OPPT’s study evaluation criteria must be substantially revised:
- OPPT should not conflate issues of reporting with study quality.
- OPPT should not discount or exclude studies for which underlying data are not publicly available.
- OPPT must consider financial conflict of interest in the evaluation of individual studies and in the evaluation of the body of evidence.
Ultimately, OPPT has developed a severely flawed “systematic review” framework. EDF urges OPPT to substantially revise its document, and subject a revised version to peer review by systematic review experts.
See our comments for more details.