CEE helps organisations commission systematic reviews and maps to achieve high standards of independence and rigour in their syntheses. Systematic reviews have become an ‘industry standard’ for synthesising evidence to inform decisions and underpin evidence-based policy and practice.

 

On this page:

  • Free resources to help with Commissioning
  • What are CEE Systematic Reviews and maps?
  • Why commission a CEE Systematic Review?
  • How to recognize a genuine CEE Systematic Review
  • When is Systematic Mapping more appropriate?
  • Relationships between commissioners, review teams and CEE
  • Tips and tricks to plan and cost a CEE Systematic Review
  • How to save time and money

Many evidence syntheses are conducted in response to the evidence needs of organisations looking to make evidence-informed decisions. An increasing number of CEE Systematic Reviews and Maps are commissioned by various organisations, institutes and agencies all over the world and many more use CEE’s free resources to help them get a product they can trust.

To get the best possible outcome from a commissioned evidence synthesis, to plan effectively the use of resources and timing of the work of the review team, make sure the review is of the best possible quality and independent of vested interest, many commissioners now work closely with CEE and make use of its guidelines and standards. Here are a few tips to help you when considering commissioning a systematic review. Don’t hesitate to contact us for more information at info@environmentalevidence.org.

 

Free resources to help with Commissioning

CEE offers a number of free resources that commissioning bodies can ask commissioned Review Teams to use to ensure standards of rigour, transparency and replicability. Think about including these in your tender documents.

  1. CEE Guidance and Standards – these help Review Teams achieve high standards a reduce risk of bias at all stages of the review process.
  2. CEESAT criteria can be used as a quick step by step guide to the reliability of reviews
  3. PROCEED – Coming soon, this service enables Review Teams to register an approved title and protocol for an evidence synthesis.
  4. CEECAT  Tool- Provides a specific guide to assessing risk of bias in studies included in the synthesis.

What are CEE Systematic Reviews and Maps?

CEE Evidence Syntheses are either Systematic Reviews or Maps as defined below.

What is a Systematic Review?
A Systematic Review collates, critically appraises, and synthesizes all available evidence relevant to a question. Reviewers use pre-defined methods to minimize bias and thus provide more reliable findings that could inform decision making.

What is a Systematic Map?
A Systematic Map collates, codes, and configures all available evidence relevant to a question. Reviewers use pre-defined methods to minimize bias and assess the extent of the evidence to provide a basis that could inform further research and synthesis.

 

Why commission a CEE Systematic Review

A key purpose of a systematic review is to provide a summary of the best available evidence to support decision making. The methodology is designed to reduce bias in collection, synthesis and reporting of evidence. A systematic review is a transparent, replicable, upgradable compilation of existing evidence from the peer-reviewed scientific and grey literature (reports, theses…) in order to produce a synthesis of knowledge on a specific question (for broader questions see evidence mapping below).

You might commission a systematic review if you

  • need an independent and rigorous summary of the evidence base
  • need to know ‘what works’, how effective a management or policy intervention is, if this effectiveness depends on some other factors (e.g. environmental or social variables, the methodology, the species or habitats targeted) or to compare several interventions and assess which one is the most effective under what circumstances.
  • want to know the impact of any policy action on other environmental variables (i.e. important side effects).
  • want to resolve a controversy where there is apparently conflicting evidence or opposing views regarding the evidence.
  • want to examine the quality of the evidence and draw recommendations about how to improve and/or orient research methodology/design to build up a stronger evidence base.
  • want to synthesize what is known about the prevalence of a disease, occurrence of a species or incidence of an event.
  • need to know the accuracy of a measurement technique or methodology and what affects it.
  • want to establish a baseline of what is known on a specific issue and how robust this knowledge is. To identify knowledge gaps or, on the contrary, fields where there is enough evidence and no further research is needed.

 

How to recognize a genuine Systematic Review

Systematic reviews comply to a pre-defined methodology. Genuine systematic reviews will either be posted in the libraries of one of the recognized collaborations (Cochrane, Campbell or CEE) or have met the standards for transparency of conduct as set out in the guidelines of these collaborations. For CEE, please peruse our Guidelines and Instructions for Authors for more details.

The protocol of a systematic review should always be made available prior to the conduct of the review itself. It should preferably be registered/published in one of the libraries of a Collaboration (Cochrane, Campbell, CEE) and peer-reviewed before the systematic review is conducted.

Publication of a protocol prior to knowledge of the available studies reduces the impact of review authors’ biases, increases transparency of methods and processes, reduces the potential for duplication, and allows peer review of the planned methods.

 

When is Systematic Mapping more appropriate

Systematic Reviews are conducted on specific questions with well defined interventions and outcomes. Often a decision maker will be faced with a broader question involving many interventions and outcomes. In this case it may be appropriate to start with a systematic mapping of the evidence. CEE Systematic Maps collects, organises and describes the evidence on a broader question without attempting to synthesise. They are used to answer how much and what type of evidence exists rather than what the evidence suggests. CEE Systematic Maps identify gaps in the evidence base as well as where a sufficient evidence exists for a systematic review and synthesis.

 

The relationship between commissioners, review teams and CEE

Many organisation will commission a systematic review using a tendering process. Commissioners may wish to consult CEE at the planning stage in order to ensure the effective use of the CEE process. We suggest that commissioners state in the ‘call for tender’ that review teams should comply with CEE guidelines and standards. We recommend that this includes the process of submission of both the protocol and full review to CEE for peer review and publication through its journal ‘Environmental Evidence‘. This provides the commissioning body with both independent assessment of standards at the start and completion of the process, and an independent open-access platform for the systematic review, demonstrating its standard and quality.

The conduct of systematic reviews is relatively new in environmental management and many potential review teams will need some training to achieve a product of the necessary standard. Our experience in training and assessing the work of review teams tells us that the quality of a review will be enhanced if there are regular exchanges between the review team and the CEE. Often this is linked to a tailored training programme to ensure that the Review Team comply with the requirements of the Collaboration.

 

Tips and tricks to plan and cost a CEE Systematic Review

At the CEE, past experience tells us that a good systematic review requires careful planning and execution. Its exact duration depends on the complexity of the review question, the skills of the review team and the type of data they will deal with.

The initial steps leading to the publication of the review protocol are similar to a project management exercise and usually take about a third of the length of the review, because a lot of work is required to anticipate all possible decisions and problems, make links with stakeholders, and get feedback from stakeholders and peer-reviewers. Sometimes during this stage it becomes obvious that the review question needs to be revisited as it is too broad or the workload will be too demanding within the decided timeframe.

The peer-review process of a protocol takes on average 1-2 months. It can be extended to 2 months for a full review (depending on its length, availability of peer-reviewers…). The Editor may ask for amendments prior to the peer-review if the manuscript does not fully comply with the instructions for authors.

Total costs of SRs vary greatly and it would be misleading to try and quote a figure, but time requirements can be estimated using software like PredicTER. By far the most substantial cost of the process is the time of skilled personnel to conduct the review. The basic CEE article processing charges, for editorial and publishing costs, are fixed and usually constitute a small percentage of the total costs.

 

How to save time and money

Our top tip for saving time and money is to ask, as part of the tender submission, that a protocol be drafted.

Question setting: contact us when you are discussing about your evidence needs and deciding whether to commission a systematic review. We can provide feedback on its feasibility, likely costs and how to phrase the review question, thus saving time and preventing false expectations.

Get a good understanding of the challenges and steps of a review by attending our introductory workshops or by organising a workshop in your workplace. See Training page.

Include in the call for tender the possibility for the review team to benefit from our training programme.