About CEESAT

What is CEESAT ? CEESAT is the CEE Synthesis Assessment Tool that is used to critically appraise each review and overview before inclusion in the CEEDER Database. CEESAT has been supported by Mistra EviEM (Sweden) which hosted a CEE workshop for a group of invited persons engaged in further improving the tool. About the CEESAT criteria in an article published by Paul Woodcock and colleagues in Biological Conservation and on the CEE website. About current CEESAT Criteria -Download CEESAT for Evidence Reviews Criteria -Download CEESAT for Evidence Overviews Criteria See here for the whole methodology of CEEDER. The CEESAT workshop group. From left: Jacqui Eales, Neal Haddaway, Ruth Garside, Nicola Randall, Barbara Livoreil, Andrew Pullin, Geoff Frampton, Christian Kohl and Biljana Macura. The CEESAT checklist provides a point by point appraisal of the confidence that can be placed in the findings of an evidence review by assessing the rigour of the methods used in the review, the transparency with which those methods are reported and the limitations imposed on synthesis by the quantity and quality of available primary data. Note that CEESAT does not distinguish between reviews that do not employ methodology that reduces risk of bias and increases reliability of findings and reviews that may have employed such methodology but do not report it. Each component of the review process is appraised according to a set of criteria. Each criteria are assigned a rating ranging from red (poor quality) to gold (high quality). Explore review components, sets of criteria, and thresholds for different ratings below. How are evidence reviews identified and rated? Step 1: We perform a systematic search of multiple databases and use search engines to collect potential environmental evidence reviews. Searches are regularly updated. Step 2: We use a set of eligibility criteria (see below) to screen potential reviews for inclusion in the CEEDER database. Step 3: Eligible evidence reviews are randomly allocated to Review College members for rating. The members rate reliability of evidence reviews using CEESAT criteria (see below).   Using CEESAT criteria as an indicator of reliability of reviews The CEESAT criteria were developed to critically appraise reviews in terms of transparency, repeatability and risk of bias. For each of 16 elements, a review is rated using four categories of review methodology as follows: Gold: Meets the standards of conduct and/or reporting that reduce risk of bias as much as could reasonably be expected. Lowest risk of bias – high repeatability – highest reliability/confidence in findings. Green: Acceptable standard of conduct/reporting that reduces risk of bias. Acceptable risk of bias – repeatable – acceptable reliability/confidence in findings. Amber: Deficiencies in conduct and/or reporting standards such that the risk of bias is increased (above green), alternatively risk of bias may be less easy to assess. Medium risk of bias – not fully repeatable – low reliability/confidence in findings Red: Serious deficiencies in conduct and/or reporting such that risk of bias is high. High risk if bias – not repeatable – little to no confidence in findings When finding a review, or comparing reviews, of relevance to your evidence needs you can either use the ratings as a whole for judging review reliability or look at certain elements that you feel are important for the context in which you are working. For example, you may feel that a comprehensive search strategy and clear eligibility criteria are crucial for you to have confidence in the findings, in which case you might want criteria 3 & 4 to be rated Gold or Green. Any Red ratings in reviews should be considered carefully to decide what impact that may have on the findings. One Red rating does not necessarily mean that you should have no confidence in the findings but it might do if that red rating is for what you consider a crucial element of review conduct (e.g. eligibility criteria). Although the categories could also be given “scores” (e.g. from 1-4), using such total scores or mean scores to compare review reliability is not necessarily meaningful and we advise against this in any context except a crude “eyeballing”. It may be more important to understand what elements of a review score Red or Amber and therefore may be deficient. Clearly reviews with ratings that are all Reds and Ambers should be viewed with low confidence, but it does not mean that the findings are wrong. At the other end of the scale reviews with ratings of mostly Gold and Green can be viewed with high confidence, but it does not mean that the findings are right. Additionally, one or two Ambers and Reds among predominantly Gold and Greens could increase risk of bias and decrease confidence substantially and therefore should be considered carefully. Finally, the CEEDER ratings are not a substitute for reading the review. There may be other aspects of the review that make it more or less useful to you as a source of evidence. Contact us: info@environmentalevidence.org Decisions on eligibility involve some subjective judgement and we will not always get it right. We welcome feedback from users on CEESAT and what is or is not included in CEEDER.

Continue reading

About CEEDER

About CEEDER The CEE Database of Evidence Reviews (CEEDER) is an open access Evidence Service started in 2018. It aims to help policy makers, managers, funders, and the general public to find reliable evidence reviews and syntheses to inform their decision making in environmental management. The aims of CEEDER are not only to guide evidence users to reliable reviews but also to provide tools to authors, editors and peer reviewers for improving the reliability of future reviews in the environmental sector (see Checklist for editors and peer reviewers). What is CEEDER as a service ? CEEDER is an open-access evidence service provided by the Collaboration for Environmental Evidence (CEE). The service collates all forms of evidence reviews and syntheses relevant to environmental management and policy interventions and anthropogenic impacts on the environment. Articles are considered for addition to the CEEDER database as they are published or otherwise become available globally. The database allows decision makers to search for and identify evidence syntheses of relevance to their evidence needs. The added value of the service is that it also provides a critical appraisal of each review’s reliability based on the primary data available for the review and the conduct of the review itself using the CEE Synthesis Appraisal Tool (CEESAT). Although primarily aimed at decision makers wishing to use evidence to inform their decisions, CEEDER will also aid in improving the reliability of reviews in the environmental sector by offering guidance and resources to authors, editors and peer reviewers. What does the CEEDER database include? The CEEDER database collates all forms of evidence reviews and syntheses relevant to environmental management, as they are published or otherwise become available globally. The CEEDER database lists available (commercially published and ‘grey’) syntheses of primary research (e.g. critical reviews, meta-analyses, systematic reviews, rapid reviews) conducted to assess evidence on a specific question of environmental policy or management relevance. Most importantly, the database provides an independent assessment of the reliability of each synthesis. Presentation of the assessment is tailored to the needs of decision makers and other evidence consumers in governmental, non-governmental and private sectors as well as the general public. To know more about how this assessment is provided please visit ABOUT CEESAT. The database contains : ‘evidence reviews’, that claim to make some form of quantitative assessment of impacts on the environment or effectiveness of interventions that would be of direct interest to management or policy, evidence overviews’, that claim to have collated and/or mapped what evidence exists on environmental impacts and interventions. Reviews and Overviews: An ‘Evidence Review’ is a review and/or synthesis (e.g. meta-analysis) of primary research findings where the objective is stated as providing an answer to a question or test of a hypothesis relating to effectiveness of an intervention or impact of an exposure. An ‘Evidence Overview’ is a review of primary research where a main objective is stated as assessing or mapping the distribution and abundance of evidence in primary studies (e.g. geographic and taxonomic patterns for identifying knowledge gluts and gaps), and/or exploring a specific topic of interest to configure bodies of evidence, on a specified question relating to effectiveness of an intervention or impact of an exposure. For more details please go to SEARCH THE CEEDER DATABASE. What is the scope of reviews presented in CEEDER? The scope of reviews included in CEEDER currently covers the whole of the environmental sector (environmental science, policy and management) and is global. The specific question or topic of the review should have relevance for policy or practice and there should be intent to synthesise primary studies and provide a measure of effect (e.g. impact of an activity or effectiveness of an intervention). What is NOT included in CEEDER? Purely descriptive reviews or ‘expert’ opinion articles are not included unless the authors claim to provide a measure of effect. The specificity of the question varies from broad global issues to precise cause and effect relationships in single species or restricted areas. Human wellbeing is included when there is also a significant environmental component in the question. Currently we specifically exclude the following subject areas except where a clear link is made to environmental management: -animal veterinary science -animal nutrition -animal behaviour -plant physiology -nutrition, improvement and growth regulation -engineering and construction -biotechnology and bioengineering -human health, education, social welfare and social justice -toxicology -species distribution and abundance (where no cause and effect is addressed). Who is behind the CEEDER Database? Contributors CEEDER is a free service provided by CEE because of the contribution of many volunteers whom we gratefully acknowledge: Funders No funders Contact Us info@environmentalevidence.org Stay Connected Sign up for our Mailing List Follow us on Twitter and LinkedIn Make a Donation CEE is registered with the Charity Commission (1157607). Visit the Charities Aid Foundation to make a donation. © 2021. CEEDER is licensed under a CC BY-SA 4.0

Continue reading