CEE supports organisations commissioning evidence syntheses to achieve high standards of independence and rigour.

On this page:

• Free resources to help with Commissioning
• What are CEE Evidence Syntheses?
• Why commission a CEE Systematic Review?
• How to recognize a genuine CEE Systematic Review
• When is Systematic Mapping more appropriate?
• How can CEE help you commission a rigorous evidence synthesis
• Tips and tricks to plan and cost a CEE Evidence Synthesis
• How to save time and money

Many evidence syntheses are conducted in response to the evidence needs of organisations looking to make evidence-informed decisions. An increasing number of CEE evidence syntheses are commissioned by  organisations, institutes and agencies all over the world and many more use CEE’s free resources to help them get a product they can trust.

To get the best possible outcome from a commissioned evidence synthesis, to plan effective use of resources and timing of the work of the review team, and ensure the review is of the best possible quality and independent of vested interest, commissioners can use CEE resources and make use of its Guidance  and Standards. Here are a few tips to help you when considering commissioning an evidence synthesis. Don’t hesitate to contact us for more information at info@environmentalevidence.org.

 

Free resources to help with Commissioning

CEE offers a number of free resources that commissioning bodies can ask commissioned Review Teams to use to ensure standards of rigour, transparency and replicability. Think about including these in your tender documents.

1. CEE Guidance and Standards – these help Review Teams achieve high standards a reduce risk of bias at all stages of the review process.
2. CEESAT criteria can be used as a quick step by step guide to the reliability of reviews
3. PROCEED – This service enables Review Teams to register an approved protocol for an evidence synthesis.
4. CEE Critical Appraisal  Tool- Provides a specific guide to assessing risk of bias in primary studies included in the synthesis.

What are CEE Evidence Syntheses?

CEE Evidence Syntheses are either Systematic Reviews, Systematic Maps or Rapid Reviews as defined below.

A Systematic Review collates, critically appraises, and synthesizes all available evidence relevant to a question. Reviewers use pre-defined methods to minimize bias and thus provide more reliable findings that could inform decision making.

A Systematic Map collates, codes, and configures all available evidence relevant to a question. Reviewers use pre-defined methods to minimize bias and assess the extent of the evidence to provide a basis that could inform further research and synthesis.

A Rapid Review follows either of the above methods but compromises standards for rapidity of production (typically less than 6 months). The compromises are specifically stated and considered as limitations when findings are reported.

 

Why commission a CEE Systematic Review

A key purpose of a systematic review is to provide a summary of the best available evidence to support decision making. The methodology is designed to reduce bias in collection, synthesis and reporting of evidence. A systematic review is a transparent, replicable, upgradable compilation of existing evidence from the peer-reviewed scientific and grey literature (reports, theses…) in order to produce a synthesis of knowledge on a specific question (for broader questions see evidence mapping below).

You might commission a systematic review if you

• need an independent and rigorous summary of the evidence base
• need to know ‘what works’, how effective a management or policy intervention is, if this effectiveness depends on some other factors (e.g. environmental or social variables, the methodology, the species or habitats targeted) or to compare several interventions and assess which one is the most effective under what circumstances.
• want to know the impact of any policy action on other environmental variables (i.e. important side effects).
• want to resolve a controversy where there is apparently conflicting evidence or opposing views regarding the evidence.
• want to examine the quality of the evidence and draw recommendations about how to improve and/or orient research methodology/design to build up a stronger evidence base.
• want to synthesize what is known about the prevalence of a disease, occurrence of a species or incidence of an event.
• need to know the accuracy of a measurement technique or methodology and what affects it.
• want to establish a baseline of what is known on a specific issue and how robust this knowledge is. To identify knowledge gaps or, on the contrary, fields where there is enough evidence and no further research is needed.

 

How to recognize a genuine Systematic Review

Systematic reviews comply to a pre-defined methodology. Genuine systematic reviews will either be posted in the libraries of one of the recognized collaborations (Cochrane, Campbell or CEE) or have met the standards for transparency of conduct as set out in the guidelines of these collaborations. For CEE, please peruse our Guidelines and Instructions for Authors for more details.

The protocol of a systematic review should always be made available prior to the conduct of the review itself. It should preferably be registered/published in one of the libraries of a Collaboration (Cochrane, Campbell, CEE) and peer-reviewed before the systematic review is conducted.

Publication of a protocol prior to knowledge of the available studies reduces the impact of review authors’ biases, increases transparency of methods and processes, reduces the potential for duplication, and allows peer review of the planned methods.

When is Systematic Mapping more appropriate

Systematic Reviews are conducted on specific questions with well defined interventions and outcomes. Often a decision maker will be faced with a broader question involving many interventions and outcomes. In this case it may be appropriate to start with a systematic mapping of the evidence. CEE Systematic Maps collects, organises and describes the evidence on a broader question without attempting to synthesise. They are used to answer how much and what type of evidence exists rather than what the evidence suggests. CEE Systematic Maps identify gaps in the evidence base as well as where a sufficient evidence exists for a systematic review and synthesis.

 

How can CEE help you commission a rigorous evidence synthesis?

Many organisations commission evidence syntheses using a tendering process. Commissioners may wish to consult CEE at the planning stage in order to ensure the effective use of the CEE process. CEE resouces and processes can also help in the following ways;

1. commissioners can simply state in the ‘call for tender’ that review teams should comply with CEE Guidance and Standards.
2. for enhanced rigour, we recommend that commissioners include the process of submission of the protocol to PROCEED for registration and
3. the evidence synthesis be submitted to CEE for peer review and publication through its journal ‘Environmental Evidence‘.

These simple steps are cumulative (you can use just step 1, 1&2 or all) and using all three provides the commissioning body with both independent assessment of standards at the start and completion of the process, and an independent open-access platform for the evidence synthesis, demonstrating its standard and quality.

The conduct of CEE standard evidence syntheses is relatively new in environmental management and many potential review teams will need some training to achieve a product of the necessary standard. Our experience in training and assessing the work of review teams tells us that the quality of a review will be enhanced if there are regular exchanges between the review team and CEE. Often this is linked to a tailored training programme to ensure that the Review Team comply with CEE requirements.

 

Tips and tricks to plan and cost a CEE Evidence Synthesis

At the CEE, past experience tells us that a good evidence synthesis requires careful planning and execution. Its exact duration depends on the complexity of the review question, the skills of the review team, and the type of data they will have to deal with.

The initial steps leading to the production of the Protocol are similar to a project management exercise and significant work is required to anticipate all possible decisions and problems, make links with stakeholders, and get feedback from stakeholders and peer-reviewers. Sometimes during this stage it becomes obvious that the review question needs to be revisited as it is too broad or the workload will be too demanding within the decided timeframe. Nevertheless this is time well spent to ensure you get the synthesis you want in the time you want it.

The registration of a protocol in PROCEED takes on average 1-2 weeks.  The Editor may ask for amendments prior to the peer-review if the protocol does not fully comply with CEE Standards. This independent check can be valuable feedback before the synthesis work commences.

Total costs of evidence syntheses vary greatly and it would be misleading to try and quote a figure. By far the most substantial cost of the process is the time of skilled personnel to conduct the synthesis. If a publication route is desired (step 3 above), the basic CEE article processing charges, for editorial and publishing costs, are fixed and usually constitute a small percentage of the total costs (read more).

 

How to save time and money

Our top tip for saving time and money is to ask, as part of the tender submission, that a protocol be drafted.

Question setting: contact us when you are discussing about your evidence needs and deciding whether to commission a systematic review. We can provide feedback on its feasibility, likely costs and how to phrase the review question, thus saving time and preventing false expectations.

Get a good understanding of the challenges and steps of a review by attending our introductory workshops or by organising a workshop in your workplace. See Training page.

Include in the call for tender the possibility for the review team to benefit from our training programme.