Best Evidence Medical Education Tay Park House, 484 Perth Road, Dundee DD2 1LR, UK BEME Coordinator: Mrs Pat Lilley Tel: +44 (0)1382 381967; Fax: +44 (0)1382 381987 Email: p.m.lilley@dundee.ac.uk http://www.bemecollaboration.org Guide for Topic Review Groups on carrying out BEME Systematic Reviews Version 2 – updated February 2003 The BEME Collaboration welcomes approaches from groups interested in conducting a BEME systematic review in a specific area. These notes are provided to help familiarise prospective groups with the BEME review process. The notes will be regularly updated to incorporate experience gained from the initial pilot reviews now under way. BEME welcomes feedback from Topic Review Groups as to how the review process may be improved. The BEME Collaboration is available to give advice and guidance to Topic Review Groups throughout the course of the review, and can offer specialist help from the BEME Information Scientist. Groups already working are accumulating valuable experience which they are pleased to pass on to new groups. The BEME website contains information on the BEME Collaboration, its structure and its activities to date. The site will be updated with a variety of information including progress made by the initial review groups. Comments and questions may be sent to the BEME Collaboration (contact details above). BEME Steering Group February 2003 1 Contents Page A B What is a BEME Systematic Review? Steps in the BEME Review Process 1 2 3 4 5 6 7 8 9 10 Selection of a topic/question for a systematic review Forming a Topic Review Group (TRG) Registration of a topic/review question The literature search Preparation of a review protocol Coding sheets and procedures for extracting evidence and data The draft report and the consultation process Final report Updating the review Prototype timetable for the review process 3 4 5 6 7 7 8 8 9 10 7 5 3 Figure I: Steps in the BEME Review Process C Training and support for Topic Review Groups Appendix I Responsibilities of the BEME Infrastructure and Topic Review Groups Appendix II Appendix IIIA Appendix IIIB How to Prepare a BEME Review Protocol Prototype BEME Coding Sheet and Notes Coding Sheet Description and Guide 11 15 Separate documents 2 A 1 2 3 What is a BEME Systematic Review? A BEME systematic review is a logical, explicit and comprehensive appraisal of available information to determine the best evidence relating to an issue in medical education. A BEME systematic review is designed to assist individual teachers, institutions and national bodies to make informed decisions about their educational practice. A BEME systematic review is a description and evaluation of evidence pertinent to a clearly formulated topic/question that uses explicit methods to systematically identify, assemble, critically analyse and synthesise information relevant to the topic of the review. The BEME strategy for review is inclusive and open to a wide range of investigative methodology. Each summary, pertinent to a clearly formulated topic/question, will be user-friendly, enabling the practitioner to judge and employ evidence according to his or her individual criteria and context. A BEME systematic review must be registered with and approved by the BEME Collaboration and is produced by a BEME Topic Review Group (TRG) following accepted BEME review procedures. 4 5 6 B 1 Steps in the review process Selection of a topic/question for a systematic review The principal emphasis in the selection of a topic/question for systematic review is a ‘bottom-up’ approach, with proposals coming from prospective reviewers. The review group will work with the BEME information scientist to define the precise scope and nature of the topic/question being considered for review. It is necessary to describe the topic/question as accurately as possible to ensure that the best evidence is retrieved and considered for the review. Points to consider are: 1.1 A BEME review topic/question should focus on and inform on practical issues or problems faced by the teacher or institution in their day-to-day practice. For example, the research topic/question should be phrased in such a way as to provide information on how the teacher or institution should respond to the adoption of a new teaching approach in their own context. BEME review questions may be most helpful where they illuminate the topic area. A valuable question will usually explore aspects of the review topic and will not necessarily be answered with a simple yes or no. The review topic/question should be precisely defined. Typically, a review topic/question will identify: • • • 1.4 population/participants e.g. undergraduate or postgraduate students the activity under investigation e.g. the timing of feedback in assessment outcomes e.g. change of attitudes or knowledge 1.2 1.3 Examples of research topics/questions for systematic reviews currently under way are: • Systematic Review of High Fidelity Simulation in Medical Education (Miami Group) 3 • • What conditions are necessary or required for assessment and feedback to enhance physician improvement? (ABIM Group) Predictive values of assessment measurements obtained in medical schools and future performance in medical practice (AGU Group) The terms used in the question should be defined: eg, What is a high-fidelity simulator? 1.5 The BEME Collaboration Information Scientist can assist by running a scoping search. These broad, simplified searches are conducted to determine the size of the body of literature relevant to the topic as initially defined. The scoping search is not comprehensive, nor does it consult all sources. Following the scoping search, it may be advisable either to broaden or narrow the scope of the proposed review topic/question in order to ensure it is manageable whilst at the same time generating sufficient studies for the review. Several iterations may be necessary before the final topic/question is defined. 1.6 2 2.1 Forming a Topic Review Group (TRG) The TRG will largely propose its own composition. The BEME Review Editorial Group (REG) strongly recommends that the following are taken into account: The TRG through its members should have: • • • • knowledge of the specific area being reviewed appropriate knowledge of medical education appropriate knowledge of research methods appropriate knowledge of the process of conducting systematic reviews It is helpful if the TRG has access to: • • • information technology skills (see 4 below, and Appendix I) clerical/database handling skills a research librarian In general it is recommended, although not essential, that the TRG has international composition. 2.2 The recommended number of TRG members will depend on the size of the review, the location of proposed group members and the resources available. It is recommended that between 6-12 is an optimum number. 3 3.1 Registration of a topic/review question Once accepted by BEME, the review topic/question and the TRG will be entered into the BEME Register of Reviews on the website. This will alert all interested parties to the proposed review. The TRG will be asked to complete a topic registration form giving: • • • names and affiliations of review group members; name and full contact details of lead reviewer; summary of the proposed review 3.2 The review question will be registered exclusively to the TRG for an initial period of one year. Within six months, BEME expects to receive a draft protocol (see 5 below) including a proposed timetable for 4 completing the review (see 10 below). If a TRG does not submit a draft protocol within the first year, and is unable to demonstrate adequate progress with a registered review, the review topic may be released to another group. 3.3 If a prospective group expresses an interest in a topic already registered and on which a TRG is actively working, the registered TRG will be asked to discuss the topic with the proposed new group to see if collaboration is feasible. If collaboration is not appropriate the proposed group will be asked to consider resubmitting a review topic/question in a related or different area. 4 4.1 The literature search The TRG should devise a robust search strategy appropriate to the review topic/question, with advice from the BEME Information Scientist: 4.1.1 4.1.2 4.1.3 The literature searches should be sensitive (broad and inclusive) in design so that no evidence is overlooked. Access to the search process should be readily available and easy to verify They should be detailed in the protocol (see 5 below) and appended to the draft and final reviews (see 7 and 8 below). Searches should be run across the various databases, specialist registers and the web; handsearching should be carried out across selected journals, the grey literature (eg, conference proceedings, newsletters) should be consulted and decisions taken on the methods the TRG plans to use to contact experts in the area [see BEME Guide on Systematic searching for evidence in medical education, to be published in April 2003). The TRG should determine the years to be covered by the search, depending on the topic area to be reviewed. Whilst a ten-year period is suggested by the BEME Collaboration, this may vary according to the number of studies in the topic area, the relevance of papers published outside the ten-year limit, etc. The TRG should justify its decision in relation to the years to be covered. It is strongly advised that a cut-off date for studies to be considered in the review should be set. The time period covered should be clearly stated in the protocol and report (see 4.1.7 below). The BEME Collaboration recommends that as far as is reasonably possible, studies should not be excluded on the grounds of either geography or language. Access to and translation of studies not published in English is the responsibility of the TRG and will depend on its resources. The BEME listserv could be used to try to locate a potential translator, or to trace a difficult-to-obtain article. The search process will involve determining the final individual search syntax for each source, executing the searches, and collating the results (removing duplicates, etc.) Depending on the time taken to carry out the review, searches may have to be rerun to locate any newly published studies that may be relevant to the review. 4.1.4 4.1.5 4.1.6 4.1.7 4.2 Initial appraisal of literature search: The TRG should assess citations generated by the search and decide, on the basis of the title/abstract, whether the citation is relevant to their topic. An evidence-based search will tend to have high sensitivity to reduce the possibility of missing evidence, therefore it will produce clearly irrelevant items that can easily be identified and eliminated. If there is any doubt as to the relevance, items should not be eliminated to ensure comprehensiveness. The fulltext article for all potentially relevant citations should be retrieved by the TRG. If it is unclear whether a citation is relevant (eg, if no abstract is provided, or if relevance is unclear from the abstract), the fulltext article must be retrieved to be properly 5 assessed. Note: TRGs have reported instances of poor quality abstracts which make it difficult to determine the relevance of the study to the review topic/question. 4.3 The TRG should decide on its policy for review of citations and coding of articles (see 6 below). BEME recommends that there should be two-person agreement on each aspect coded. In the event of difference of opinion, a third person should decide. A record of differences of opinion should be kept. One TRG has reported poorer inter-rater agreement on qualitative data than on quantitative data. The final inclusion and exclusion criteria should be agreed by the TRG and stated in the protocol (see 5 below). 4.4 5 Preparation of the review protocol The protocol describes the review's context and objectives, defines the review's topic/question and remit, and outlines the methodology to be employed to critically appraise and analyse evidence from the selected studies (see Appendix II). 5.1 5.2 5.3 5.4 Within an agreed period, usually a maximum of 6 months from registration of the review question, the TRG should submit a draft review protocol. The draft protocol will be reviewed by the REG, who may submit it for peer review as appropriate. The draft protocol will be approved by the REG, possibly following revisions by the TRG if required. The approved protocol will be posted on the BEME website. 6 Coding sheets and procedures for extracting evidence and data The coding process identifies those studies that will ultimately be used in the review, and those that will be excluded if they do not meet the scope of the review or quality standards, according to the criteria set out in the protocol (see Appendix II). Excluded studies and the reasons for their exclusion should be recorded and this information should be appended to the final review. 6.1 A prototype coding sheet is available to all groups for adaptation to meet the needs of their review in terms of the data to be extracted (Appendix III). Adaptations made to the prototype coding sheet by the TRG should be documented. Initial groups have reported the need for extensive changes to the BEME prototype coding sheet to meet their needs. A BEME working group will be set up to look at this aspect, based on input from TRGs. When the TRG finalises its coding sheet an electronic form will be provided by the BEME Collaboration for use by the TRG, linked to a customised database specific to the TRG and provided by the BEME Collaboration. If TRGs wish to use their own database they should check its compatibility with the Dundee-designed database to avoid future problems of consolidation of data. All studies retrieved by the systematic searches will ultimately be stored in the TIMELIT database which will form a repository for studies generated through BEME reviews. 6.2 6.3 7 7.1 The draft report and the consultation process Following a rigorous and objective synthesis of the evidence base the TRG will produce a draft report with its preliminary conclusions, which will be submitted to the REG for review. 6 7.2 7.3 The draft report will be posted on the BEME website for wider comment. Following approval by the REG, the draft report will be presented for discussion at either a specially convened working group or at a session at one or more international meetings (eg AMEE or AAMC, and in some cases at medical specialty meetings) and feedback relating to possible modifications will be requested. The aim is to identify any relevant information that may have been missed from the review, to provide input on the interpretation of the evidence and to comment on the user-friendliness of the review. 8 8.1 8.2 8.3 8.4 Final report Following the consultation process detailed in (7) above, the final report will be prepared by the TRG and submitted to the REG for approval. The approved review will be published on the BEME website, under the authorship of TRG members. The TRG is encouraged to prepare papers based on the review’s findings for submission by the authors to relevant journals. The BEME Collaboration must be acknowledged in all papers submitted for publication. The database of studies relevant to the review will be available via the BEME website. 9 9.1 9.2 Updating the review Reviews should normally be updated within three years of the date of the review. The TRG will have first option of updating the review. If the TRG is unable to provide resources for an update, any other group may request permission from BEME to update the review. 10 Prototype timetable for the review process The steps in the BEME Review Process are given in Figure 1. Whilst each review will vary, the following is the anticipated duration and recommended maximum for each stage: Anticipated duration Forming TRG and defining question: Registration of topic: Literature search: Preparing protocol: Data abstraction: Draft report: Final report: Updating review: Recommended maximum 4-6 weeks ) 2-4 weeks ) 6 months from 2-4 weeks ) registration 1-2 months ) 2-4 months ) 2-3 months 1-3 months 8 months from ) protocol ) 4 months from draft report 3 years from 7 final report C Training and support for Topic Review Groups BEME aims that Topic Review Groups should become self-sufficient in terms of developing the expertise needed to conduct BEME systematic reviews. Support and advice is readily available as follows: • • • • the BEME Steering Group will advise on matters relating to BEME policy generally; the BEME Review Editorial Group should be consulted on matters of policy specifically relating to conducting systematic reviews; the BEME Information Scientist may be contacted for advice on issues relating to literature searches; TRGs already operating are available to pass on the valuable experience they have gained to date. Printed resource materials A BEME Guide Systematic Searching for Evidence in Medical Education by Alex Haig is scheduled for completion in April 2003 and will then be made available to Topic Review Groups. The guide outlines the challenges presented by searching for evidence in medical education and comprehensively covers relevant databases, websites and internet searching, and grey literature. Workshops Workshops are run at major medical education meetings (eg, AAMC, AMEE, ASME) on practical aspects relating to conducting systematic reviews. A one-day pre-conference workshop ‘Finding and appraising evidence in medical education’ is being run by Alex Haig and Marshall Dozier at the AMEE Bern Conference (www.amee.org). Training workshops may also be run either by the BEME Information Scientist or other members of the BEME Steering Group at the specific request of a Topic Review Group at a location of their choice. Because of limited resources, Topic Review Groups must allocate funding for a Training Workshop. 8 Figure 1 Steps in the BEME Review Process Submission of Outline Proposal by Topic Review Group Acceptance by Review Editorial Group Entered in BEME Register of Reviews on BEME website Submission of Protocol including Review Topic/Question Approval by Review Editorial Group Entered in BEME list of approved Protocols on website Draft Report of Systematic Review Consultation Process Final Report of Systematic Review Entered in BEME Register of Reviews on BEME website Publications Updating of Review 9 Appendix 1 Best Evidence Medical Education (BEME) Responsibilities of BEME Collaboration infrastructure and Topic Review Groups (TRGs) in relation to systematic reviews A Responsibilities of BEME Collaboration infrastructure 1 1.1 1.2 1.3 General Identification of role of the systematic review within overall context and philosophy of BEME Consideration of proposed methodology for systematic reviews by the BEME Review Editorial Group (REG) Preparation and updating of BEME documentation: • • • • Information for TRGs on how to conduct a BEME systematic review How to construct a BEME protocol Responsibilities of BEME infrastructure and TRGs Prototype coding sheet 2 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 2.10 2.11 Specific to each review Recommendation on composition of TRG Recommendation on choice of topic and review question Assistance from Information Scientist with scoping search Registration of topic as official ‘BEME’ review, and group as an official BEME Topic Review Group Provision of training to groups, where requested and as funding permits Provision of advice on budget for proposed review Advise on coding sheet for review, to be developed from BEME prototype Advise on defining protocol and approval of protocol Provision of BEME database for TRGs to enter data from all studies identified in literature search Review draft report prepared by TRG and provide comments where appropriate Agree final report. 10 3 3.1 3.2 Publicity and publication Disseminate information about BEME review topics and TRGs as appropriate Liaise with TRG with regard to publication of the review in appropriate formats, including on the BEME website. B 1 1.1 1.2 1.3 1.4 1.5 2 2.1 2.2 2.3 2.4 2.5 2.6 Responsibilities of Topic Review Group Prior to conducting systematic review Identify local members and leader for TRG Collaborate with BEME REG to identify international reviewers if appropriate Formulate the research question for approval by the REG Establish communications process between group members and agree requirements for face-to-face or ata-distance meetings for duration of review Identify funding for the review, taking into account the issues listed in Appendix A. Implementation of systematic review Refine question and revise coding sheet on basis of scoping search of titles and abstracts of articles provided by Information Scientist; Prepare review protocol for submission to REG for approval Access full-text of articles selected by TRG for review Code articles as per protocol and enter information into database provided Synthesise evidence and prepare draft report for approval Prepare final report for publication on BEME website. 11 Appendix A (appended to ‘Responsibilities’ document) Issues for Topic Review Groups (TRGs) to consider in estimating budget for BEME systematic review 1 Staffing costs and overheads: • • • Funding for review group members, information officer, secretarial, administrative and data input as appropriate, for the duration of the review; Consultancy costs for expertise not available within the TRG, eg statistician Overheads relating to services provided by institution, eg heating, communications network, space, central administration. 2 Database searching/document acquisition/translation costs: • • • Depending on the extent of the review question this could result in a considerable number of articles to acquire through inter-library loans. Articles published in foreign languages: translation costs may be incurred if articles in languages not spoken by review group members are identified in the search. Cost of accessing databases: whilst access to some databases is free, others impose a charge. Accessing full-text articles: TRGs will need to access the full text of articles selected for review. 3 Communication costs: This will depend on • • • • the number of group members and their location the duration of the review the perceived need for face-to-face meetings other means of communication. eg teleconferences, mailing costs 4 Equipment and consumables: • • • Computers and software Printing and photocopying Office supplies (continued) 12 5 Group training costs: • Funds for training workshop, if required. 6 Contribution to BEME infrastructure costs: • Funding for staff support is at present provided by NHS Education for Scotland (computing and information services), and the Association for Medical Education in Europe (administration). A contribution to central funding from the American Board of Internal Medicine is gratefully acknowledged. Additional funding is required for further development of BEME. It is hoped that review groups, where funding permits, will make some contribution to infrastructure costs. • 13 Appendix II Draft BEME Review Protocol Rationale for BEME Review Protocol The protocol details the procedures for identifying, evaluating and amalgamating the evidence for a BEME review. The protocol describes the review's context and objectives, defines the review's question and remit, and outlines the methodology to be employed to critically appraise and analyse evidence from the selected studies. Setting out an explicit and thorough protocol before the review commences defines the review process and will help ensure that the evidence to be considered is systematic and comprehensive and that the possibility of bias is diminished. When a topic for a systematic review is proposed by a Topic Review Group (TRG) and accepted provisionally by the BEME Review Editorial Group (REG), the proposed title, summary and group contact details are posted on the BEME website. The aim is to prevent duplication of effort, and also to identify others who may wish to collaborate in the review. The TRG is then asked to produce a protocol, set out as detailed below. A draft protocol is submitted to the REG for comment. The approved protocol, following revisions if requested by the REG, is posted on the BEME website for wider comment. Note: This document should be read in conjunction with Guide for Topic Review Groups, and Responsibilities of BEME Topic Review Groups and the BEME Infrastructure (Appendix I) Components of a BEME protocol The protocol comprises the following: 1 2 3 4 5 6 7 8 9 10 11 Cover sheet Background to the topic Review question(s) and objectives Search sources and strategies Study selection criteria Procedure for extracting data Synthesis of extracted evidence Project timetable Conflict of interest statement Plans for updating the review Changes to the protocol Details of each component are given below. 1 Cover sheet The cover sheet should include the following information: • • Review title Names and affiliations of group members, with lead reviewer highlighted 14 • • Contact details for the lead reviewer Sources of support 2 Background to the topic The review topic should be explained and placed in context to make the rationale and need for the review explicit. It should contain: • • • a brief overview of the research question a conceptual discussion relating to the research question a statement of the significance of the research question Reference should be made to any previous reviews of the research topic where these exist and reasons for the need for the current systematic review. 3 Review topic/question(s) and objectives A BEME review topic/question should focus on and inform on practical issues or problems faced by the teacher or institution in their day-to-day practice. For example, the research topic/question should be phrased in such a way as to provide information on how the teacher or institution should respond to the adoption of a new teaching approach in their own context. The review topic/question should be precisely defined. Typically, a review topic/question will identify: • • • population/participants e.g. undergraduate or postgraduate students the activity under investigation e.g. the timing of feedback in assessment outcomes e.g. change of attitudes or knowledge BEME review questions may be most helpful where they illuminate the topic area. A valuable question will usually explore aspects of the review topic and will not necessarily be answered with a simple yes or no eg. What are the aspects of high fidelity simulators that promote effective learning? The terms used in the question should be defined: eg, What is a high-fidelity simulator? Having defined the question it may be necessary to broaden or narrow the search (see 4 below). 4 Search sources and strategies BEME reviews require systematic searches for evidence across a comprehensive range of sources relevant to the topic. Systematic searching in medical education is particularly challenging given that the evidence is widely dispersed and very often poorly indexed. An appropriate search strategy should be constructed to ensure the remit of the review is met and this will be published as an appendix to the completed review. The TRG should identify the full range of sources to be consulted including databases and specialist registers to be accessed, journals to be hand-searched, the grey literature to be accessed and methods that will be used to contact experts in the field. The TRG should indicate the time limit for the search (see Guide for Topic Review Groups, Section 4). Neither language nor geography should restrict searches. Depending upon the timetable for the review (see 8 below), it may be necessary to include in the protocol the provision to conduct updating searches during the review. 15 5 Study selection criteria The TRG should make explicit the criteria it plans to use for selecting or excluding studies in order that the results are objective and reproducible, bearing in mind the following: 5.1 5.2 5.3 As part of the BEME approach, studies should not be excluded solely on the basis of type of study design; Where studies are to be excluded for reasons of methodology, these reasons should be made explicit; If TRGs intend to set different criteria with regard to time limit, geography or language from the recommendations given in the document Guide for Topic Review Groups, Section 4, these should be detailed here. 6 Procedure for extracting data The prototype BEME coding sheet (Appendix IIIA) is available to TRGs to adapt to meet the needs of the group, and adaptations made should be documented. The TRG should include a statement of how many group members will code each article, how the TRG intends to check for inter-rater reliability and the proposed procedure to resolve differences in coding of studies. Changes made to the recommendations given in the document Guide for Topic Review Groups, Section 6, should be detailed. 7 Synthesis of extracted evidence The methodology used in BEME systematic reviews will inevitably evolve during the initial pilot reviews. TRGs should make explicit the methodological procedures they will use to synthesise the evidence. 8 Project timetable TRGs should submit a proposed timetable for the stages involved in producing the systematic review. A prototype timetable is available for consultation ( see Guide for Topic Review Groups, Section 10). Timetables will vary between BEME groups, depending on an individual group's resources, its size, geographic distribution of members and complexity of the topic, etc. 9 Conflict of interest statement The protocol must include a conflict of interest statement, relating to all TRG members. These include academic, institutional, political, personal or other conflicts. Special concern must be given to financial conflict of interest, with the acknowledgement of any financial support, including money, hospitality or subsidy. 10 Plans for updating the review TRGs should indicate if they intend to update the systematic review after completion and if so, include brief details and a potential timeline (see Guide for Topic Review Groups, Section 9). 11 Changes to the Protocol 16 Whilst a well thought out protocol should reduce the need for changes, unanticipated issues may arise and modifications to the review topic/question, coding sheets and/or protocol may become necessary as the TRG becomes more familiar with the evidence. Any subsequent changes to the protocol must be carefully recorded, with the reasons for the changes and the date the changes took place. Significant changes to the protocol must be submitted to the REG for approval. BEME Collaboration March 2002 17