MEDEV

School of Medical Sciences Education Development

Introduction

OCTAVE (Optimising Computer-Aided and Traditional Assessment in Veterinary Education) is a three-year HEFCE funded project, led by the Royal Veterinary College, on behalf of the four English veterinary schools, in collaboration with the two Scottish veterinary schools, designed to rationalise veterinary assessment schemes.

The project has two main aims: to maximise the use of valid computer-aided assessment in formative and summative examinations, and to update traditional practical examinations through the development of a structured, objective approach to assessing practical skills.

The project commenced in earnest in March 2003, with the appointment of Dr Stan Head as the project manager. Early contact with staff at the 4 English veterinary schools showed continued enthusiasm for the project. However, on the CAA side, a major practical hurdle was noted; the requirement to gain software skills for authoring the questions using Questionmark Perception as the database manager and assessment software.

Computer aided assessment

This has been solved in two ways. We have developed a method which allows authoring in MS-Word, using an OCTAVE template, and direct importation into the database. Our template allows all text formatting, and inclusion of a picture. With this method, there is no requirement for the naive author to learn new software skills.

Alternatively, we have developed a web page through which a single question can be written and sent by email to OCTAVE. Without any further processing, these web-authored questions can also be imported into the database.

We are now at the stage where questions are trickling into OCTAVE from individuals. In order to encourage and create a communal atmosphere for question-writing, Focus Group Meetings have been instituted where taught modules are blitzed. Staff meet over a good lunch (provided by OCTAVE) and questions emerge. More of these activities are planned.

In addition, in order to overcome the difficulty of writing educationally instructive feedback, which is essential for the potential of the database to be fully realised by students in formative or self assessment mode, we are piloting sessions where authors talk through questions with a student and the conversation is recorded. The content of this discussion acts as a basis for feedback.

Clinical examination

The second strand of the OCTAVE project is the development of Objective Structured Clinical Examination (OSCE) style assessments for veterinary students. OSCEs have been used for many years in medical assessment andwe have adopted a similar format for veterinary students. A number of five-minute scenarios have been written, and these were used for the first time at the Royal Veterinary College (RVC) in January 2004.

The scenarios are clearly structured so that each student receives exactly the same task, with the same starting situation in terms of instructions and layout of any models and equipment.

The assessment is objective; that is, it is a checklist of whether a candidate has performed a task or not, and a scenario comprises between 12 and 23 important steps.

Assessment in practice

With training, naive assessors can be used for many stations, although some stations do require specialist knowledge in order to assess the student fairly and competently. In order to distinguish veterinary OSCEs from steeplechase-style spot tests, which are essentially paper-based, short answer questions, the scenarios have been designed to assess the candidate's ability to perform real practical tasks.

A graduate from veterinary school, unlike many other professional graduates, has, on their first day after graduation, to have mastered a number of practical competences, named euphemistically Day One skills. These have been determined, and published by the professional statutory body, the Royal College of Veterinary Surgeons (RCVS (2001) Veterinary Education and Training: A Framework for 2010 and Beyond). The OSCEs that we have developed and used in a formative manner at RVC test at least some of these Day One skills.

Members of all the UK veterinary schools were invited to attend the OSCE assessment in January 2004 and the representative from Liverpool veterinary school both attended and acted as assessor for a number of stations. A number of workshops on writing successful OSCEs have been held and more are planned at the RVC. The workshops are hosted by Dr Katharine Boursicot, who has run training workshops for OSCE examiners at Barts and the London School of Medicine. The focus of the workshop is to write OSCEs and, in the same session, trial what has been written.

As the scenarios are for practical skills, a comprehensive and rapid system for collecting the required equipment has been built up. Even a cow's pelvis and calving snares were sourced and moved to the workshop classroom within 20 minutes. A representative from Cambridge Veterinary School contributed to the last workshop.

Standard setting

Methods of setting the pass/fail level for OSCE stations have been investigated; these include those that use norm referencing and those based on criterion referencing.

As a consequence of our results and the practical skills format of the assessment, we have found it necessary to develop an approach to standard-setting based on the borderline group method. The results of those students who have just failed, together with the results of those students who have just passed, are used to set a pass/fail level. Our modification answers some of the criticisms of the borderline group method as originally described.

The future of OSCEs at the RVC

The final examination in March 2005 will have OSCE spot tests components, in addition to traditional written papers. In order to allow the students experience of this form of assessment, OSCE's are being arranged at the end of the monthly rotations for the fourth year students and will occur every fourth Friday, commencing in August 2004.

Initial student feedback on OSCEs shows considerable enthusiasm, and recognition of the validity of this type of test.

We look forward to a busy and rewarding period at the RVC and the other English veterinary schools as OSCEs become a valued additional assessment resource.