Section 3

Planning the conduct of an Evidence Synthesis

Last updated: July 3rd 2023.

3.1 Options for engaging with CEE

CEE resources for planning, registering and conducting an evidence synthesis are free for authors, funders and other stakeholders to use. There are 3 basic levels of engagement with CEE.

1. Use CEE resources but otherwise have no engagement in terms of registering, publishing or meeting CEE Standards. In this case we simply hope that the use of CEE resources will help you improve aspects of your evidence synthesis.
2. Use CEE resources and register your Protocol in PROCEED but then have no further engagement in terms of publishing and meeting CEE standards. Registering your Title and Protocol in PROCEED is free (including feedback from Editors) and will help you plan your evidence synthesis to a high standard.
3. Fully engage with CEE to register and publish your evidence synthesis to CEE standards. This will involve registering your Protocol in PROCEED and publishing (subject to peer review) your evidence synthesis in Environmental Evidence. The latter involves an Article Processing Charge (see the journal website for latest details on price and potential waivers). Authors committing to the CEE process also have the option to submit their protocol for publication in Environmental Evidence (subject to peer review and in addition to registration in PROCEED). This option will involve a separate Article Processing Charge.

3.2 Planning an evidence synthesis to CEE Standards

To meet CEE standards for the conduct of evidence syntheses the Review Team will need to establish an a priori protocol detailing how they will conduct each stage of the evidence synthesis. The protocol sets out how the question was formulated and how each stage of the synthesis will be conducted and is submitted for registration in PROCEED in advance of conducting the synthesis. The initial steps that aid in planning the conduct are described in subsequent sections. Here we describe some initial planning steps followed by guidance on the structure of the protocol itself.

Click here for a summary of CEE Standards for conducting and reporting. In addition, a set of checklists (ROSES) that can be used during the preparation of any systematic evidence are available at https://www.roses-reporting.com/ .  These checklists indicate the correct level of detail to be reported in protocols and synthesis reports so that the high standards of replicability are achieved. The detailed planning of each stage of the evidence synthesis can be found in the relevant sections below and should be consulted when preparing the protocol.

3.3 Scoping the evidence

Before the commencement of an evidence synthesis, it is essential that some ‘scoping’ is undertaken to guide the construction of a comprehensive and appropriate protocol, and to provide an indication of the likely form of the synthesis and thus facilitate resource planning. In certain circumstances, it may not be efficient to commit to a synthesis without some prior estimation of its value in terms of the likely extent and reliability of its findings. In addition, when scoping a Systematic Review, an estimate of the type of data (quantitative, qualitative) will be needed to inform the type of data synthesis that might be appropriate.

Scoping may be undertaken by the commissioning organisation, by the Review Team, or a combination of the two. A thorough scoping exercise might entail:

• The development and testing of a search strategy (see below).
• An estimate of the volume of relevant literature and the volume of material likely to be unavailable in easily-accessible format (see below).
• An estimate of resources required based on the above, including time and personnel to achieve the search and sorting of the literature, possible financial resources to obtain some articles, contact some authors, use the help of translators, and even plan for possible need of statisticians if quantitative data are identified during this scoping stage.
• An estimate of the study types likely to be found (as identified through focused data extraction of a small subset of relevant papers). This may indicate whether a meta-analysis will be possible (for Systematic Review only).
• An estimate of possible impairments and limitations in the conduct of the synthesis (access to literature, language limitations, need for experts…) to try to find solutions or be transparent about limitations.

The expected output from a scoping exercise is an estimate of the quantity of evidence, and a characterisation of the likely evidence base, pertaining to the question. Scoping may lead to reconsideration of the review question and scope to meet the available resources. The extent of investment in scoping required to meet CEE standards will differ with each evidence synthesis. Planning of specific stages of a synthesis is covered in later sections.

3.4 Estimating resource requirements

Whilst the process of scoping may seem like a time-consuming one, the benefits can be considerable and this early investment will allow the development of a comprehensive protocol as well as improve the focus and efficiency of the review. Scoping should provide an estimate of the timeline of the review and team effort required so that a realistic budget can be prepared or the likely costs compared with the available resources. Claims that rigorous evidence syntheses inevitably take a long time (years rather than months) is largely a myth. It is true that many evidence syntheses do take years to complete, but this is almost always due to lack of sufficient planning, synthesis skills, or limitations of resources, and not due to the methodology itself. With good planning (including adequate resources) and a skilled Review Team, alongside a committed stakeholder group, evidence syntheses can be completed in months and this time is decreasing as more tools are becoming available to help speed up key stages, such as screening.

3.5 Writing and registering a CEE-standard protocol

3.5.1 Purpose of the protocol

CEE evidence syntheses require the registration of their protocol (project plan) as an independent document, before the synthesis is conducted (See Section 3.1). There are several reasons for doing so:

• Data are not evidence unless accompanied by a protocol and analysis.
• Within the CEE approach the title and protocol act as a formal registration of intent by the Review Team to conduct a CEE evidence synthesis on a given topic. It allows CEE to inform the scientific community of this project, and to let the Review Team know about any prior similar projects, or ongoing ones.
• The protocol acts as an a priori guide and reference to the conduct of the synthesis that reflects views of stakeholders and that the Review Team and their commissioners agreed upon during the planning stage (including the scoping exercise).
• The protocol is essential to minimise reviewer bias (e.g. resulting from ad-hoc decisions made or ‘mission creep’ during the synthesis process) and make the review process as rigorous, transparent, and well-defined as possible.
• The protocol enables explicit and compulsory recording of any change that may occur during the conduct of the synthesis that would not have been foreseen. This is particularly important to ensure the confidence of the consumers, commissioners and stakeholders about the reasons for changes.

3.5.2 Developing and writing a protocol

To develop and write a protocol that encompasses all aspects of the subsequent (planned) evidence synthesis, specific sections are required in order to guide the authors and make sure all components of the project have been carefully considered.

The protocol’s background section should present the problem being addressed and the rationale for why the evidence synthesis is required. Where possible, a ‘theory of change’ or conceptual model should be presented that explains the process(es) whereby the intervention or exposure factor is thought to have an impact or cause a change in the subject population (see www.theoryofchange.org/what-is-theory-of-change/). In more complex situations a proposed causal chain, linking intervention(s) to outcome(s), may be helpful. The structure of an evidence synthesis protocol mirrors the structure of the Systematic Review or Systematic Map that it guides. Beside a formal presentation of the question and its background (the “real world” context), a protocol sets out (informed by the scoping process – see above) the strategy for searching for relevant studies and defines eligibility criteria for article screening. The question elements defined in the question formulation stage provide the a priori inclusion criteria important for the objectivity and transparency of the synthesis. They should also lead to a description of the kinds of evidence (e.g. study designs) that you would consider valid to include in the synthesis. An evidence synthesis protocol should also detail, with rationale, the methods to be used for eligibility screening, data coding/extraction, study validity assessment, and data synthesis, and state any conflicts of interest including details of any funding sources.

Since the protocol sets out what the synthesis aims to achieve, it is useful for getting the engagement of experts who may have data to contribute. Anyone reading the protocol should clearly understand the nature of the question and what type of evidence will inform it. Registering and posting of protocols (e.g. in PROCEED) provides transparency and also acts as a record of which syntheses are in progress, enabling others to see if a synthesis is being conducted that may be of interest to them, or to prevent the initiation of a synthesis on a topic that is already underway. For examples of recently completed protocols, visit PROCEED and the Environmental Evidence Journal.

Once an evidence synthesis protocol has been registered and published as final, changes are discouraged. However, it may become necessary during the course of an evidence synthesis to make revisions because of deviations from the proposed methods. These changes should be clearly documented within the final synthesis report so that transparency and repeatability can be maintained. If a major change is necessary to a protocol part-way through conduct of an evidence synthesis (e.g. change of question or major change in scope) then the protocol should be updated on the registration platform, and the change should then be applied to all references, or studies, as appropriate, to avoid introducing bias. The final evidence synthesis report should explicitly state how the final Systematic Review or Systematic Map methods differed from the protocol.

Protocols are plans of conduct and can rarely be fully comprehensive. They are judged in this context during the registration and CEE peer review processes. Consequently, the acceptance and publication by CEE of a protocol does not guarantee acceptance of the resulting synthesis report. Problems with the latter may occur due to conduct that was not mentioned or not fully transparent in the Protocol.

As a general rule, a protocol should set out the plan for a single report (a Systematic review or a Systematic map). Exceptionally, where there is a strong logical case made, a single Protocol may set out a plan for multiple reports. This should be anticipated and fully explained in the protocol and should not be a post-hoc decision. Whether the multiple report route is permissible will be decided by the CEE Editorial Boards and is a trade-off between the efficiency provided by the publication of one protocol and the legitimacy and feasibility of combining several evidence syntheses together.

3.5.3 Format for CEE Protocols

Templates for registration of protocols are available in PROCEED. The format for submitting protocols to Environmental Evidence are similar and can be found on the journal website by following the links below.

For Systematic Map Protocols

For Systematic Review Protocols

 

Continue to Section 4 – Conducting a search

Return to the Table of Contents