A systematic review of the literature on the effectiveness of self-assessment in clinical education

N.B. The information below is authored by the mini-project applicants, not by staff of MEDEV. This text represents the views and opinions of the mini-project team only, not those of MEDEV or its affiliates.

Principal investigator

Brian McKinstry,

Full list of project partners

Brian McKinstry
Rachel Adams
Helen Allbutt
Gellisse Bagnall
Iain Colthart
Sheila Edward
Alison Evans
Alex Haig
Susan Hrisos
Jan Illing
Heather Peacock

Background

Our group hope to undertake a systematic review of the literature on self-assessment in clinical education using the Best Evidence Medical Education (BEME) protocol. Our specific research question asks:

What is the evidence that explicit methods used in self-assessment

a) identify learning needs?

b) promote change in learner activity?

c) promote change in clinical practice?

The BEME Collaboration (1) is a group of individuals and institutions who are committed to the promotion of Best Evidence Medical Education through the dissemination of information which allows medical teachers, institutions and all concerned with medical education to make decisions on the basis of the best evidence available, the production of appropriate systematic reviews of medical education which reflect the best evidence available and meet the needs of the user.

Self-assessment is being increasingly used as a formative tool both in undergraduate and postgraduate clinical education and is frequently used as a means of evaluating learning interventions. However, the ability to self assess is seldom tested and self-assessment skills are rarely taught. In a review in Academic Medicine (2) in 1991, Michael Gordon concluded:

Provide a brief description and rationale of your proposed mini-project e.g. What are you going to do? Why do you want to do this mini-project? Why is this mini-project worth funding? (500 words maximum)

Our group hope to undertake a systematic review of the literature on self-assessment in clinical education using the Best Evidence Medical Education (BEME) protocol. Our specific research question asks:

What is the evidence that explicit methods used in self-assessment

a) identify learning needs?

b) promote change in learner activity?

c) promote change in clinical practice?

The BEME Collaboration (1) is a group of individuals and institutions who are committed to the promotion of Best Evidence Medical Education through the dissemination of information which allows medical teachers, institutions and all concerned with medical education to make decisions on the basis of the best evidence available, the production of appropriate systematic reviews of medical education which reflect the best evidence available and meet the needs of the user.

Self-assessment is being increasingly used as a formative tool both in undergraduate and postgraduate clinical education and is frequently used as a means of evaluating learning interventions. However, the ability to self assess is seldom tested and self-assessment skills are rarely taught. In a review in Academic Medicine (2) in 1991, Michael Gordon concluded:

"In the absence of specific training, self-assessment of knowledge and performance appeared to be an artefact of the learner's global self-concept. Self-assessments remained remarkably resistant to the influence of external evaluative information such a supervisory reviews, grades or examination scores."

In his review of over 1000 papers citing self-assessment as the focus of the paper he found only 13 papers which explored systems of improving self assessment per se.

We hope to review this work and also to see if the evidence base has changed since 1991.

We have a group of eleven from several clinical and social science disciplines, from Edinburgh, Newcastle, Glasgow and Leeds who are interested in taking part in the review. Our information scientist will perform the initial searches and these will be augmented with hand searches of derived references and a review of 'grey' literature. To begin with we have kept our search wide including non-clinical sources, but, depending on the initial results, may narrow this down to clinical education studies. The method employed (described by Reeves) involves deriving a clear research question, developing a recording framework, identifying suitable papers, sending these to pairs of reviewers, evaluating the papers against the framework and finally synthesising the results. Quality assurance procedures will draw on the experience of previous BEME reviews and the expertise of other systematic reviewers to ensure a rigorous approach. Unlike other systematic reviews such as Cochrane there is seldom a sufficient number of high quality randomised trials to allow meta-analysis and so a more inclusive approach is taken which considers other types of study, for example, cohort studies or qualitative studies.

References
1. BEME Collaboration http://www.bemecollaboration.org/

2. Reeves S. Koppel I. Barr H. Freeth D. Hammick M. Twelve tips for undertaking a systematic review. Medical Teacher 2002; 22: 358-63

3. Gordon,M.J A review of the validity and accuracy of self-assessments in health professions training. Academic medicine 1991; 66: 762-9

Proposed activities

June 2004 - Clarify boundaries of systematic review

July 2004 - Initial literature search examples sent to project members

September 2004 - Meeting to discuss initial review in broad terms, confirm boundaries and design recording framework

Oct-Nov 2004 - Systematic literature search. Abstracts sent to participants

Nov 2004 - Meeting to confirm recording framework and scope of review

December 2004 - Likely useful papers identified and full papers obtained

Jan-April 2005 - Full papers reviewed against recording framework and hand searched of derived references for further papers.

May-July 2005 - Synthesis of findings and correspondence with acknowledged field leaders to ensure coverage

July-December 2005 - Writing of BEME review and papers

Proposed outcomes

We hope the whole report will be available on the BEME website, we hope to publish several papers in peer-reviewed educational journals and to present the findings at national and international conferences. We would hope that the results of the review might inform a wider audience of successful, evidence-based methods or tools for self assessment or suggest areas of future research. We also anticipate benefits in our own institutions for example further development of formative learning portfolios for undergraduate and postgraduate doctors.

Proposed expertise

The project team brings considerable expertise to this task. Alex Haig our information scientist is a veteran of several BEME reviews. Brian McKinstry, Gellisse Bagnall, and Jan Illing have all published widely on educational topics and all three have experience of research project management in clinical education. The other members of the team have backgrounds in nursing, medicine, education and social science. All have expertise in research and critical review, recently enhanced by a 2 day systematic review workshop with Prof Marilyn Hammick the national expert in this field

Expertise of grant holder and project team

The project team brings considerable expertise to this task. Alex Haig our information scientist is a veteran of several BEME reviews. Brian McKinstry, Gellisse Bagnall, and Jan Illing have all published widely on educational topics and all three have experience of research project management in clinical education. The other members of the team have backgrounds in nursing, medicine, education and social science. All have expertise in research and critical review, recently enhanced by a 2 day systematic review workshop with Prof Marilyn Hammick the national expert in this field

Similar work

Searches of Timelit, Omni and LTSN databases revealed several small scale uni-professional reviews of varying quality since Michael Gordon's in 1991 which will provide a useful starting point. However, there has been no recent high quality large scale systematic review. No other group has listed an interest in this area with BEME.

Contact details

Amount awarded: 3000

MEDEV project contact: Megan Quentin-Baxter

Reports and resources

  • Effectiveness of Self Assessment
  • Interim report
  •  
     
    MEDEV, School of Medical Sciences Education Development,
    Faculty of Medical Sciences, Newcastle University, NE2 4HH

    |