Writing and registering a Protocol
Last updated July 4th 2018
4.1 Purpose of the Protocol
CEE Evidence Syntheses require the publication of their Protocol (project plan) as an independent document, before the synthesis is conducted. There are several reasons for doing so:
- Data are not evidence unless accompanied by a Protocol and analysis.
- Within the CEE approach the Protocol acts as a formal registration of intent by the Review Team to conduct a CEE Evidence Synthesis on a given topic. It allows CEE to inform the scientific community of this project, and to let the Review Team know about any prior similar projects, or ongoing ones.
- The Protocol acts as an a priori guide and reference to the conduct of the synthesis that reflects views of stakeholders and that the Review Team and their commissioners agreed upon during the planning stage (including the scoping exercise).
- The Protocol is essential to minimise reviewer bias (e.g. resulting from ad-hoc decisions made or ‘mission creep’ during the synthesis process) and make the review process as rigorous, transparent, and well-defined as possible.
- The Protocol enables explicit and compulsory recording of any change that may occur during the conduct of the synthesis that would not have been foreseen. This is particularly important to ensure the confidence of the consumers, commissioners and stakeholders about the reasons for changes.
IMPORTANT: COMMITMENT TO REGISTER AND PUBLISH WITH CEE. By registering and publishing your Protocol with CEE you are registering your intent to conduct, and submit to for publication, a CEE Systematic Review/Map. You will be asked to confirm that you and your co-authors are aware of and agree with this commitment when you submit your Protocol for publication in Environmental Evidence.
4.2 Developing and writing a Protocol
The Protocol’s background section should present the problem being addressed and the rationale for why a Systematic Review or Systematic Map is required. Where possible, a ‘theory of change’ or conceptual model should be presented that explains the process(es) whereby the intervention or exposure factor is thought to have an impact or cause a change in the subject population (see www.theoryofchange.org/what-is-theory-of-change/). In more complex situations a proposed causal chain, linking intervention(s) to outcome(s), may be helpful. The structure of an Evidence Synthesis Protocol mirrors the structure of the Systematic Review or Systematic Map that it guides. Beside a formal presentation of the question and its background (the “real world” context), a Protocol sets out (informed by the scoping process – see above) the strategy for searching for relevant studies and defines eligibility criteria for article screening. The question elements defined in the question formulation stage provide the a priori inclusion criteria important for the objectivity and transparency of the synthesis. They should also lead to a description of the kinds of evidence (e.g. study designs) that you would consider valid to include in the synthesis. An Evidence Synthesis Protocol should also detail, with rationale, the likely methods to be used for eligibility screening, data coding/extraction, critical appraisal (Systematic Review only), and data synthesis, and state any conflicts of interest including details of any funding sources.
Since the Protocol sets out what the synthesis aims to achieve, it is useful for getting the engagement of experts who may have data to contribute. Anyone reading the Protocol should clearly understand the nature of the question and what type of evidence will inform it. Registering and posting of Protocols on the CEE website provides transparency and also acts as a record of which syntheses are in progress, enabling others to see if a synthesis is being conducted that may be of interest to them, or to prevent the initiation of a synthesis on a topic that is already underway. An example of Protocol development is given in Box 4.1. For examples of recently completed Protocols, visit the Environmental Evidence Library at: https://environmentalevidence.org/reviews-in-progress.
Once an Evidence Synthesis Protocol has been peer-reviewed and published as final, changes are discouraged. However, it may become necessary during the course of an evidence synthesis to make revisions because of deviations from the proposed methods. These changes should be clearly documented within the final synthesis report so that transparency and repeatability can be maintained. If a major change is necessary to a Protocol part-way through a Systematic Review or Systematic Map (e.g. change of question or major change in scope) then the Protocol should be updated in consultation with CEE, and the change should then be applied to all references, or studies, as appropriate, to avoid introducing bias. The final Evidence Synthesis report should explicitly state how the final Systematic Review or Map methods differed from the Protocol.
Protocols are plans of conduct and can rarely be fully comprehensive. They are judged in this context during the CEE peer review process. Consequently, the acceptance and publication by CEE of a synthesis Protocol does not guarantee acceptance of the resulting synthesis report. Problems with the latter may occur due to conduct that was not mentioned or not fully transparent in the Protocol.
As a general rule, a Protocol should set out the plan for a single report (a Systematic review or a Systematic map). Exceptionally, where there is a strong logical case made, a single Protocol may set out a plan for multiple reports. This should be anticipated and fully explained in the Protocol and should not be a post-hoc decision. Whether the multiple report route is permissible will be decided by the CEE Editorial Board and is a trade-off between the efficiency provided by the publication of one Protocol and the legitimacy and feasibility of combining several reviews or maps together.
Box 4.1 Example of Protocol Development
4.3 Format for CEE Protocols
The format and template for submitting protocols can be found on the Environmental Evidence website by following the links below