MEDEV

School of Medical Sciences Education Development

Introduction

In common with other total philosophies, total assessment is concerned with the acquisition of the maximum amount of data involved in any assessment process, in order to facilitate better feedback for students and statistical information for staff.

The paucity of feedback in medical education has been questioned by
Ende,1 and explained first as the failure to obtain the data. In the past
minimal amounts of feedback were provided to students, for instance
from an essay they may have had written comments and a grade-2 data
items. Using a total assessment, the individual criteria, the grade and the
written comments are each scored and recorded, so giving perhaps 20
data items per assessment, a tenfold increase per student. From the staff
point of view, a wealth of data can allow decisions to be on the direction
of students’ progress, based on performance indicators in particular
criterion areas. Similarly, with MCQstyle tests, the information provided
as feedback to students can be equally valuable to staff across examined
topics, to direct areas of weakness for whole student groups. With the
assessment of practical or physical aspects of the assessment process,
such as the OSCE, the scoring criteria are designed in a way that facilitates
the acquisition of large amounts of data quickly, as timeliness is also an
important aspect to this process. OSCEs have been used for both
summative and formative assessment, commonly with taped examinations1.
The significant improvement in student performance, has been demonstrated in the use of the OSCE,2 and increasing feedback can
only reinforce this knowledge ands kills gain.

For students the outcome of all of these procedures is improved feedback, which is intended to inform and assist them in self-diagnosing their weaknesses and directing their learning. For staff, the outcomes should be faster turnaround of results,improved amounts and availability of data, and less work in the coordination and processing of data.

From a quality perspective, the capture, interpretation and retention of large amounts of data allows the long-term monitoring of individual student performance and course consistency (with a consequential improvement in both areas). Increasingly, courses are focused on larger numbers of lower value assessments, instead of a single major make or break final at the end. This gives scope for total assessment to acquire large amounts of data.

Methods for the fast and simple acquisition of large amounts of data fall generally into two different methods:

• Direct acquisition; this can be performed by either the direct capture of assessment data into spreadsheets of databases or by the use of systems which perform electronic assessment, such as computer-based assessment systems.

• Indirect acquisition; this is the transcription of assessment data by staff, from a paper-based assessment, or capture of data optically from forms or similar.

In Swansea a combination of both of these methods is used within the
Graduate Entry Programme in Medicine. Essays have been marked and marks transcribed, which is now being trialled for direct entry of assessment marks into a database by academic staff. On-line assessment through QuestionMark Perception is used, OSCEs and Exams are performed with OMR forms and data acquired using Remark Office. In a recent Mock OSCE exercise, to prepare students for their Intermediate MB, the estimated  amount of data went up by a factor of 10, and was performed across manned and unmanned 12 stations. This information was processed and a 25 page feedback reports returned to the student in less than 1 week. This has been received very positively by the students, as borne out by their evaluation of the exercise.

Data storage can be a significant issue with the student data, as it needs to be available in a format that allows access and processing of the data, but secure enough to prevent misuse or unauthorised access. In the case of indirect acquisition, the data has to be entered into a spreadsheet or database from the original script, or captured optically and exported from the original software. With direct acquisition, the student data can be stored directly into the student records database, to enable a direct connection to the data processing. It should also enable simple access by staff, facilitating their interpretation of the data.

To begin the process, and in order to provide good quality formative feedback to students quickly, in the first instance we took the decision to
use an MS Word Mailmerge from an MS Excel spreadsheet. The mailmerge documents were then printed and handed back to the students in a sealed envelope, as there were about 25 pieces of paper per student and University regulations stipulate the return of assessment results only to the named individual.

In the initial phase, the acquired data has been stored in an Excel spreadsheet for ease of analysis; and has been fed back to the students as
printed, mail merged Word. Initially this seemed to be the best method, as
it is quick and visible, and experience has shown that students have a
preference for paper-based feedback. Further developments intend to
favour on-line methods of assessment result collation and feedback, with a
current database-driven system in development, with the intention of
interfacing this with our VLE (BlackBoard), to provide an on-line record, or possibly mail merged and returned electronically, to form part of a student’s PDP.

Current feedback statements are literal text, based on the criterion against which the students are judged i.e. 1. Wash/decontaminate hands. Stand/sit on patient’s RHS. Performance - Inadequate. It is intended in future that the feedback will be an interpreted version of the criterion statement, for example the above statement would be fed back as 1. Your performance on washing and decontaminating your hands and sitting on the patients RHS was judged by the examiner as inadequate. In future ensure that you follow simple hygiene and courtesy before approaching the patient.

This type of interpreted feedback was successfully piloted with a group of
first year Pharmacy students,3 based on the students performance in a
summative, computer-based assessment, which had been studied from an eLearning simulation of a Pharmacology experiment. It is envisaged that this type and form of feedback will by applied to all areas of assessment within the GEP course, including future OSCEs, written assignment and computer-based testing. The next phase of OSCE work is directed towards direct acquisition of OSCE performance data, using tablet PCs or computers equipped with touch-screens. Using this interactive method should considerable simplify and speed up the acquisition of the data, additionally the examiners can be prompted to complete forms, if criteria should be unmarked.