PORSCHE external evaluation
An independent evaluation report commissioned by the PORSCHE project funded under the HEA/JISC Open Educational Resources Programme (UKOER) Phase 2
Date: October 2011
Authors: Helen Beetham and Yolande Knight
Introduction
This report forms one of the deliverables of the UKOER phase II PORSCHE project. 'Pathways to Open Resource Sharing through Convergence in Healthcare Education' set out to promote seamless access to learning resources across clinical and academic settings, specifically by establishing a long-term partnership and sharing of educational content (or at least of content records and access arrangements) between Jorum (UK HE sector) and the National e-Learning Repository (NeLR) (UK NHS). A second objective was to achieve widespread uptake of the Risk-Kit OER Toolkit – developed with earlier funding for use in HE settings – across a range of NHS organisations, by aligning it with the NHS e-learning readiness toolkit. Similarly, it was hoped to embed into clinical practice some of the recommendations and model practices that had been developed for academic settings, covering copyright clearance, open licensing, and obtaining of patient consent. The project also set out to promote debate over the development of a Consent Commons, and establish the value of all these outcomes to enhancing the student experience in both academic and clinical placement settings.
Critical issues for the project, as highlighted in the final report, were:
- delays in establishing a basis for bidirectional sharing of learning resources between Jorum and NeLR due to technical issues outside the control of the project;
- significant differences of culture, infrastructure, and e-learning resource (including time and expertise) between NHS Trusts and relevant academic departments.
The independent evaluators were not asked to report on the first of these issues in any detail. This study focuses instead on the experience of participants and close contacts of the PORSCHE project, as reported in interviews and reflective accounts. It was commissioned from an independent evaluation team to provide an external perspective on the value, impact and benefits of the project at the end of its funded life.
Methods
The evaluators identified a number of key evaluation questions by mapping the UK OER Release Strand synthesis/evaluation framework against the stated objectives of the PORSCHE project, taking into account the modified Kirkpatrick framework which the internal evaluation team had been using to collate and assess evidence throughout the project's lifetime. These questions, and the associated mapping, are separately available as a synthesis framework with a comprehensive mapping of project outputs, including the evidence contained in this report. The key evaluation questions are used to structure the findings reported below.
The evaluators carried out semi-structured interviews with ten contacts identified by the PORSCHE project team. Contacts were interviewed by telephone against an agreed schedule, the interviews recorded and transcribed, and scripts approved by the interviewees before analysis. Analysis was thematic, both independently of and subsequently mapped to the evaluation questions. The evaluators also analysed twenty video clips of participants at the e-Learning in Health Conference 2011, the text of three case studies, and reflective SWOT analyses carried out by the project team in relation to their own activities.
Findings
1. What are the main motivations for OER release and re-use in medical education?
All interviewees described producing openly licensed content as a personal ambition but few were actually taking this forward. As a result, discussion of motives and barriers was not always anchored securely in personal experience. Motivations for releasing content included the following:
Altruism and professional pride:
I have always wanted to get the best value and at the same time the aim of always getting the students to be the best that they can. That seems to be through sharing rather than the commercial model.
Giving back to the profession I work in.
Responding to the zeitgeist:
We wanted to be involved [in OOER] as it was timely.
Better to be a leader than a follower.
This is the way to go. In five or ten years' time things will automatically be open. Publishers are being naïve – books aren’t the way forward. Getting out a book is a bizarre thing for students to do. The iTunes model is the way forward.
Previous positive experiences of sharing:
We built e-learning which was shared amongst all Scottish institutions... and that was a fantastic experience at the time. [It was] a good test of 'can we share?' and the answer in Scotland was 'yes'.
This same interviewee noted that positive experiences and direct benefits from a small-scale sharing relationship can lead to sharing on a broader front: 'If we can deal with [our local rivals] then we can deal with Scottish Universities, then the UK, then world!'
Efficiencies in content production:
Good resources for medical education are costly and technically difficult to produce, which makes sharing and re-use more attractive.
People are doing video demonstrations and I don’t necessarily have the ability to create those.
It's about us as a community sharing stuff – what’s the point in reinventing the wheel 7 times?
People are increasingly pushed for time and are recognising that there is a lot of really good stuff out there on the web being shared under Creative Commons.
However, funding constraints are also squeezing still further the resources for in-house content development:
It’s very hard to get hold of funds, and so collaborative work is even more important than ever. But it’s trying to find the time and create those networks and those links, to develop the projects and work on that.
This competes against other calls on our time.
So, paradoxically the short-term costs of building capacity to develop shareable content, and generating inter-organisational trust and commitment, may simply not be covered, putting the potential savings of shared content out of reach.
2. What are the main barriers to OER release and re-use in medical education?
The first barrier to engagement with the OER agenda, both in terms of OER release and re-use and irrespective of discipline, is one of awareness. Interviews with participants at the ‘eLearning in Health 2011’ conference acknowledged this issue:
I think a lot of academics are still oblivious to the copyright legislation and think “I can still Googlesearch something and use that image” and aren’t aware of it. There’s some staff development work still to do, and when you make them aware legitimately that they can reuse resources and save them time, they are really up for that.
Once users have been made aware of the potential to share, reuse and release content, structural barriers become more significant. Several were picked up in the course of this evaluation.
Costing models, and the desire to see a return on investment in content production:
I can see us being asked why we are giving away IPR
Thinking that we are going to be giving it away for free - I think that is something that maybe all institutions are grappling with
There is still a perception in some parts of the NHS, particularly with the drive for Trusts to become financially independent, that income be generated from the development of e-learning content
In fact when this issue was explored in more detail, none of the interviewees believed there was a viable business case for monetising content within an NHS market, but all felt this case had not yet been made at policy level. There are similar arguments in the University sector as it enters a time of funding constraint and search for new revenues, but institutions are more likely to focus on knowledge transfer and knowledge services than on content per se.
Clearing copyright and consent:
As in other educational sectors, concerns arise around copyright clearance and the use of third party content. However, in medical education there are additional factors. Specialised content such as video or simulations are often produced by or in partnership with commercial ventures who may have retained elements of copyright, while images may have been sourced from the NHS Photo Bank which licences images for use within the NHS only.
A lot of the time we don't own the content in the first place. I'm thinking of a specific example. In [x region] we developed an e-learning package … we could share the package freely but because it turned out we hadn't bought the royalties to the video we had to pay the supplier to make that available to anyone else.
In addition, patient consent to use content for teaching and learning may have been obtained only for limited uses, or may not have been adequately recorded.
Patient confidentiality and IPR make making [resources] globally open more challenging.
Copyright just seems such a complex issue, that people would just rather not think about it at all
'Life and death' standards of content reliability
Content released in medical education contexts must be reliable and up to date if it is going to be made publicly available, because of the risk of harm to patients.
As a subject matter expert it is your name on the line, particularly with medical resources, you could say something that could have big consequences when shared out.
How you could guarantee that all that content was OK to go there? … especially when you go on a site [iTunesU] that is completely open?
A culture of closed systems (though this may be changing):
There is no culture of open licensing in the NHS.
Ideally we'd like to produce packages that other people could freely use. But sometimes it's challenge enough to get it to work within our own systems.
These cultural and systemic issues were highlighted not only in practice situations (eg the NHS), but also within HE institutions, despite the ideas of collaboration and sharing seeming to be better established:
[institutions] like to make sure we are in competition with each other, we don't always want to collaborate unless we can see some definite benefits from working together.
However, it was acknowledged by some that this culture may be changing:
In our profession until quite recently everything was quite secret. Things seem to have changed and that’s not the way I work. If someone asks me for my communication skills materials, I say yes, and can I have yours too!
Note that even in this positive scenario, 'sharing' is still imagined in terms of reciprocal exchange rather than open release.
3. How far did the project achieve its aim to showcase OER in healthcare education in clinical and academic contexts?
To a limited extent only, due to the delay in establishing the metadata interface between Jorum and the existing medical repositories. However, there was evidence from the interviews that practices of content production were changing in both clinical and academic contexts, specifically around the inclusion of more third-party open content (especially images), the use of more in-house development, and negotiating contracts with external suppliers that purchased rather than licensing the IPR on content developed.
4. How far did the project achieve its aim to raise awareness of consent issues?
To a significant degree. All interviewees, whether telephone, video or through event feedback form, described learning something from their encounters with the PORSCHE project. All felt that they had become far more aware of consent issues, and many were considering the development of policy in this area. However, it was generally felt that this was a long-term goal, and that retrospectively clearing consent for images, video etc to be openly released would be prohibitive.
There is a willlingness from the [NHS] photo library to share now, but they realise that to do that they would have to go back to square one, approach all the people in the images, approach the original photographers, buy the royalties. So realistically it's not practical to retrospectively do that.
The consent commons is just too big an issue for us to tackle at this time... having a consent commons policy in place would require a lot of work to set up, and to manage that, and perhaps retrospectively go back to all our images and video content and apply it.
5. How far did the project achieve its aim to acknowledge the value of sharing within and across contexts?
To some extent the project was preaching to the converted, but interviewees felt that their commitment to sharing was reinforced through contact with the project. Structural barriers to sharing have been explored in question 2 and are explored further in question 7. They include: the need or perceived need to track users within NHS content systems; a focus on compliance in core materials; unclear copyright arrangements; the challenges of clearing consent; and complex, integrated resources that can be difficult to localise or repurpose.
Participants in the interviews and at the e-learning in Health conference identified a clear benefit in sharing practise and process with others, and reflecting on practise in their own context.
'[We] discovered the lack of cross-departmental communication, which has meant lots of people doing interesting things in isolation'
Just a general awareness of what other organisations are using to find free resources. It's good to chat to other people in the same situation, see where they're getting images from.
So a lot of the things I got out of the workshop were formalising things we have been doing for some time, but actually putting policies in place.
Many of those people reached and encouraged by the project are key change agents in their own department or trust.
What I'm trying to do is to encourage colleagues to use creative commons content and then we can focus on developing resources which aren't currently available, and then share those back to the wider educational community.
6. Have project resources supported practice change in the direction of more OER release and reuse?
Everyone interviewed described learning from their encounters with the PORSCHE project and its team, invariably about copyright and consent, and often about other issues in open release.
'We learned an awful lot. The big picture really helped us anticipate and visualise the issues... we were going to have to address.'
'Gave us good coherent structure we had to go through... it was developing a process we didn’t have and giving us a greater insight into how the process should be formed.'
'we tried to work out the name and swim lanes – what do I do? Who do I ask? What if contributors work in different places? Who sends email? Where is responsibility?'
'We integrated the tools we needed with localised processes we needed to go through.'
'I was able to take away some very good exemplar disclaimer wording that I’ve used... I also hadn’t ever seen the Google Advanced search, so I’ve used that a lot. And then also the Xpert tool that was demonstrated - I hadn’t really seen that... So those three were all very worthwhile.
I suppose the other big thing I took home from that which I am now working on for our organisation is a take-down policy. Again its probably something we've done informally but we've never documented it.
Specific materials that had been actively used and embedded in participants' own contexts were the model consent forms, Risk-Kit toolkit, Medivac toolkit, workflows, and the Xpert tool – which is not a PORSCHE product but shows the project successfully embedding 'best of' OER resources from other developers. Interviewees generally felt more confident in using third party content, so that they could mitigate the legal risks. This included using medical images and videos from YouTube.
We quickly realised there are a few things on YouTube we can use, so now it's just about contacting the relevant people to see if we can embed it into one of our programmes
Interviewees were in roles where they were able and expected to cascade their learning to others, and all had done so to immediate colleagues, ranging from two to twelve in their team. Several had embedded the learning in new guidance, especially on use of third party content and sourcing of images, and in take-down policies. Three were organising local workshops or meetings to pass on the learning, which they expected to be attended by 12 to 30 members of staff. Others were developing eLearning materials which would be influenced by the ideas PORSCHE had passed on, and these would be accessible to large numbers of staff - potentially up to 90,000. All of these interviewees represented groups of people (departments or teams) who themselves were responsible for delivering learning, training or content in subjects allied to medicine, so the potential for cascade was very high. For example, as a direct consequence of attending a PORSCHE event, the University of Essex is looking to use an adapted version of the Medivac consent form across all departments (rolled out from medicine) and several NHS trusts are modifying their copyright guidance. However, there was a perception that communications from the PORSCHE project were still somewhat too oriented towards HE, and needed to be simplified for the NHS where 'we are not so far down the road' of sharing content.
Interview scripts and feedback responses reveal the journey that individuals take in their awareness of licensing issues, from ignorance (if an image is available on Google I can use it) to awareness (there's a problem), often accompanied by fear (I could be sued). Many participants described their initial awareness of these issues in terms of 'fear', 'worry', 'concern' and even 'shock'.
'I feared the threat of breaking copyright and any penalties that may be brought'
'Shock that I can't just use any image I find online and that I could be fined thousands for doing this'.
I think there is a bit of fear, because as a subject matter expert it is your name on the line, particularly with medical resources... and possibly fear of using technology
Critical to whether that person goes on to create/share content safely and confidently is how the fear is managed. Three interviewees considered there was a risk that people would be put off rather than encouraged to release content, because of the complexity of the messages about IPR and consent.
It was very scary to hear all the checks that need to be done, and you start to think 'what's the point?' People might well be put off producing content at all, when we actually want more people to be doing this.
However, interviewees conceded that most people knew there was an issue, even if they chose to ignore it, and were looking for the 'reassurance' that they were doing the right thing. So resources should emphasise the comfort of knowing a practice is safe rather than the possibly grave consequences if practice is not legal. The journey from fear to reassurance needed simple maps and guides, accessible principles, and easy-to-use tools. One said 'it has to be as easy as a right click to save'.
There's some staff development work still to do, and when you make them aware that legitimately they can reuse resources and save time, then they are really up for that.
7. What are the specific issues involved in releasing/sharing medical content?
Some issues identified in these interviews and feedback materials were found in academic and clinical settings (such as the resource-intensive nature of checking copyright), but very specific differences were also highlighted.
In Higher Education 'everything is rightly curriculum driven'. Because of the need to demonstrate that students have covered specific aspects of the curriculum, authentication, tracking, and local/personal feedback are critical. These are difficult to reproduce in open content, as highlighted in the Risk-kit and in case study 2. For example:
Feedback won't [work] for students not on [our] campus. I would like to think e.g. Newcastle could use our resources, but they would have to be reconfigured locally.
You can add assessment in QuestionMark Perception, which the university has a license for, which means the students can be tested on their knowledge, and we can get cohort-based reports that everyone has passed that knowledge - so from a fit-for-practice audit purpose we can do that.
Even those who described themselves as champions of sharing and who had attended open content workshops made the assumption that content would be authored in closed systems such as Moodle and QMP. Having (the perception of) control over the resources produced and the students accessing them was seen as critical from a risk management and compliance perspective:
The stuff that came out of Nottingham is quite good on resuscitation etc, but now it’s old, because they’ve changed the number of repetitions, they’ve slightly changed how to do it, so things do get old, and you need to be able to update them.
If it goes in a repository and you change it, you need to remember where every word is, but if it’s behind a log-in, you can’t link people directly to it.
We had issues with students from other universities getting access to our VLE and it dissuaded some lecturers from putting content up on there.
Inside the NHS the situation is even starker, and access to content is fraught with technical as well as pedagogic constraints.
NHS firewalls hinder access to open content and reusability in teaching.
There are still a lot of differences across technological platforms... it isn't a transparent system.
Different virtual learning environments create barriers to sharing content.
Tracking e-learning within NHS IT systems can be mightily complex in fact. Because of the lack of commonality of IT systems and LMSs that we use.
[We] in the NHS are far behind having the IT infrastructure, open architecture and wi-fi capability to make use of [open content]
There is also a strong compliance culture and fewer resources for innovative development: 'We don’t really have the time to develop content from scratch and we also don’t have the huge budgets that we used to have or money to go out and procure content'.
Sharing content within the NHS was perceived to be happening on a wider scale, thanks to regional initiatives such as content clubs and partnerships, and the national repository. PORSCHE was cited as a third element within this trend. Relations with external suppliers were also being conducted from the basis of a better understanding of licences and their implications.
When we work with e-learning content now, we write into the content with the supplier that we will own all the IPR at the end of the development, whereas historically we almost bought a license to the content and allowed the supplier to take the IPR.
This tentative progress could easily be reversed, however, if the transition to Foundation Trusts leads senior managers to believe that content can and should be monetised. There is even a risk that the trend towards in-house development of eLearning could be reversed, forcing trusts to commission expensive materials from commercial suppliers on a local basis.
The idea of sharing content beyond the NHS proved difficult to discuss with interviewees. Few avenues other than content clubs or the national repository had been considered even by the project contacts.
We are willing to share the content that we create via the national NHS e-learning repository.
The content we develop is really by the NHS, for the NHS, in terms of addressing workforce training needs.
In both academic and clinical contexts, checking copyright clearance on legacy content was found to be resource-intensive and technically demanding, even without the additional burden of clearing consent. This could make retrofitting content for open release almost impossible:
It was a matter of identifying people who we knew- that we had either created it all from scratch ourselves, or that we knew would be ok to send out and about there and then, getting the content contributor’s permission.
[I learned that] it's hard to change existing stuff, but creating new materials is easy once you make that choice not to use anything that’s not CC licensed'
Perhaps even more strongly than in other subject areas there was a perception of local (Trust/department) and subject specialism, militating against both re-use and release;
I do think that there is a tendency for content experts to want to make their own- that’s just a human nature thing, they think they would maybe do it differently, or it’s not all of the content they want, or it’s not covering the content they want, and also about making it local.
We have x hundred different e-learning packages on handwashing because we all think our way is the best and only way. And actually being able to adopt and adapt what has been used elsewhere in the NHS can be fraught with difficulty because of subject matter experts not agreeing with each other on what's right.
What I’m doing with the health professionals is so specific to a set of problem-based learning triggers that we use, that I don’t know it will be that useful unless someone was going to take the whole package of triggers.
A lot of the content is aimed at acute Trusts, it’s not easily breakable, so I can’t localise that content for my Trust
The NHS 'core curriculum' and mandatory training materials created further barriers to open release, for example because the packages were difficult to disagreggate or customise.
We’ve received a package that the sender would be quite keen for us all to use, but yes, it is just so complex... and it can’t be taken apart without the source files.
There's national content available that we can't modify or obtain. We've got some access to it but it's embedded into national learning materials and we can't put our own policy on there or do things like that – we can't modify them for what we need.
Also the quality and imprimature of the NHS materials gave rise to fears that locally-produced content could not achieve a professional enough finish.
There’s no real reason why people couldn’t take the stuff, but it doesn’t look professional enough. We’re creating it within lessons within Moodle and we don’t have a huge budget for it. It’s not looking slick like the core learning unit stuff [and...] the core learning unit won’t let us use that unless we pay through the nose
It's the quality of the content in the repository. We wouldn't mind being associated with it, I just always think there should be some kind of quality control.
Technical problems with the repository interface pilots are dealt with in more detail in the project Final Report. Participants in these interviews were more concerned with the basic functionality of Jorum or the NHS e-Learning Repository (only one participant, who was very closely involved with the technical aspects of the project, commented on both).
What has happened to JORUM is really nice to see. It’s very simple to use,
You don’t to have to be technical at all, [Jorum] is actually quite plain to use.
I've tried a few times to put our content onto the NHS e-learning repository but I've struggled a bit. I'm not a big fan.
It's hard to find items on [NeLR] because there's no advanced search options
More useable tools and repositories/services were often requested. However, it was anticipated that the digital capabilities of staff and students would continue to be a factor.
we still have huge issues with staff’s technical abilities - the training of staff to use the technical software hasn’t really happened, and it isn’t just in my university, it is across the board.
it is still quite technical, so … I find subject matter experts, unless they are early adopters or highly technical, find it very difficult to engage in creating resources.
Although eLearning is becoming much more expected by students and more widely appreciated by teaching staff, educational and academic organisations are still developing the capability to offer it appropriately
8. What new quality criteria are important when releasing materials for medical education?
9. How accessible are the OERs released/shared to potential users?
10. How reusable/repurposable are the OERs released/shared?
Due to the small number of OERs released during the funded period of the project, there is limited evidence concerning these three questions. We can say, however, that project contacts are now more aware of quality and design issues around sharing and reuse.
Traditionally what has happened is people say 'I want to make an e-learning application on the heart', we say 'OK, say why and how e-learning is going to help'. We go through a well-defined process. Now there are additional processes. Who is the audience and potential audience? Is it of relevance to a global healthcare audience? Do we have your permission, Professor X, to release this to the world?'
[I'm happy to release content if..] I know that I’ve made the content, I’ve cleared content, and I’ve got patients’ consent, I’ve got the actors’ consent, I know where the images have come from or I’ve taken them.
There was some limited consideration of the new audiences for content once it is openly released, but as discussed above this was very limited. Release of medical content in iTunesU or YouTube was not regarded as a priority or even as very desirable.
People are looking at the iTunesU part of it as more of an advert, a marketing strategy- they’re not looking at it as a learning strategy.
Resources are inherently more reusable if consent has been cleared as part of the development process, as case studies 1 and 3 illustrated. There is clearly a significant impact that the project can have on the reusability of medical resources as discussions about Consent Commons progress.
'they’ve achieved a lot, even just with their patient consent form, and making everyone aware that you do need the consent of everyone involved in video and audio, and just keeping those records and giving yourself the reassurance that everything you are doing is in the best practice
More resources are being produced that are disaggregable, but only by those with considerable technical expertise:
It’s a different methodology of development- it perhaps means for us narrowing down to assets, individual assets like videos and images and coupling that with text and making it breakable and shareable.
As noted, images and video were seen as particularly valuable for reuse and the PORSCHE workshops provided opportunities to cascade awareness of what was available:
It's good to chat to other people in the same situation, see where they're getting images from. … We would like to use more free images in our content and reference them properly.
11. What organisational issues arise for academic departments and for NHS education services when open release is undertaken?
As discussed, there are a range of organisational barriers to open content release, experienced both by academic departments – around compliance, student tracking, and the reliability of information – and by NHS Trusts where only one or two people may be tasked with developing resources.
Resourcing and business case: in HE there were concerns about the weakness of the business case relative to the time invested.
We still need to make the business case
Senior managers do not understand the value of developing and releasing/sharing materials, nor that developing quality materials that meet legal requirement takes a lot of time.
Time is the usual barrier. Creating things of a standard where you are happy for them to go global will take more time to create than something of less polish. If resources are to be seen by lots of people you'll want them to be very polished. If it is just for your students you can do a more off the cuff, face to face, blended session.
Branding, and raising quality for the purposes of external 'showcasing', add further time penalties. Case study 3 highlighted this issue from a reuse perspective, but interviewees were more concerned with the implications for development and release.
The Universities want to brand it as they hold the copyright as you are an employee, and marketing and external relations will want to have a part in it which means it will take longer to do.
In the NHS, lack of investment in content can lead to a sense of relative poverty and an ethos that 'sharing begins at home'.
Currently our first priority is to make sure the content is right for [this Trust], and then if other organisations come to us I'm happy for them to have the link.
The difference in the culture between NHS and HE is that we just – although we've got educators working in the NHS, most of the people working in education roles have a job to do, have hundreds of people to deliver training to, have no time or budget, and not the luxury of spending time on development of e-learning.
Restructuring of the NHS has brought additional problems:
The SHAs days are numbered, and creating new posts in the NHS is just impossible at the moment.
All NHS trusts by 2013 have to become foundation trusts. One of the requirements of that is to become income generating, working as a commercial organisation rather than historically how we've worked. So there is a bit of a conflict over the willingness to share something openly versus the need to generate income.
12. What new strategies, policies and guidelines are being produced to support open release?
An important output of the project has been the analysis of NHS Trust policies and guidelines on sharing, copyright and consent. The project identified that not all NHS policies were accessible, and that both content and implementation vary widely. This means that the tension between sharing and commercialisation pressures can play out very differently in different regions and organisations. As a result of this review and its recommendations, national frameworks are likely to become better coordinated and more favourable to open approaches. As noted in the Final Report, project impacts have included:
- a revised NeLR checklist and process linked to the toolkits developed by the project (mentioned by one interviewee)
- revised NeLR takedown policy and disclaimer, now being replicated by contributors e.g. Yorkshire Ambulance Service (mentioned by two interviewees)
- guidance on Creative Commons disseminated widely within the medical education community (mentioned by several in general terms)
Most participants in the interviews were in the early days of producing new policies but said that contact with the PORSCHE project had provided an impetus as well as valuable tools, exemplars, guidance etc.
At the University of Essex a new policy on consent, developed for the Medical School is being considered for roll-out across the whole University because of the expertise the team has been able to gather in this area.
I have the ability to take it university-wide, because people do take notice of what I have to say... We do deal with consent all the time.
We’re going to start reviewing our consent forms and our procedures to make sure we're doing it the right way. ...The Trust employs 10,000 staff and they all have learning needs and can access our content.
IPR policies are also being updated, and workshops organised with teaching staff and with staff who are involved in the development of content. Three participants mentioned the development of new take-down policies:
[the main message for me] was to do with a take-down policy, that we definitely need one in place just in case third party images – upstream copyright – becomes a problem.
13. How can institutions support staff to develop the relevant skills?
Although not highlighted as a formal aim and objective of the PORSCHE project, the development of new forms of support, guidance and staff development has occurred as a result of achieving the official aims and objectives.
Most participants in PORSCHE events themselves run staff training and/or staff development workshops, with a number stating that they were intending to disseminate their learning from PORSCHE events to their colleagues through these pathways. In addition, two approaches to supporting staff were in evidence.
The service model involves a central support unit helping in the development of resources and provision of guidance:
'We encourage medics and dentists to come to us as we are developing material, and we go through the tools with them. They come into the medical unit and we go through the tools and process, we brief them as to the intention of making materials open.'
The secondment model involves opportunities for academics and practitioners to develop their own expertise through time away from their normal role, usually in a general e-learning context, though implicitly with an exposure to OER development and release through contact with people who have been influenced by PORSCHE.
'we are fortunate to have clinical teaching fellows, who are taking a year out to get experience of e-learning in some cases. Then they go back to practising medicine. I want [them to] give us ideas about how the process has been for them and what would make the process easier'
Several participants also mentioned a need for highly usable tools to allow staff with lower levels of technical skill to engage in release and reuse.
[there needs to be] more emphasis on the tools that are required to get content developed. That part of the puzzle is missing... more products, more repositories, or more guides that are very simple to use, almost like a Web 2.0 version, away from the jargon, a lot of plain English and some hand-holding is what I think is very important. A lot of the time we are going out and dealing with staff who aren’t technical, that haven’t got a library background and would not know about some of the issues like metadata and organising content, and structures and interoperability and SCORM, and you mention these things and people are going to run a mile.
Some mentioned that they had developed their social media skills as one aspect of getting across the messages about reuse and resource availability:
I didn’t really understand Twitter at all – now I understand hashtags and how they work I get loads of interesting things.
The same participant noted that different social media might be used to target students as end-users of open content, and this chimed with a small number of comments about the need to raise students' awareness and skills if open content is to be used by them directly:
Everyone should have the knowledge that I'll take home today, and that will include students
However, even with easy-to-use tools and more nuanced guidance, time and expertise remain critical to the production of high quality, shareable content. Direct support for individuals, whether through workshops, centrals services or secondments, seems to be essential for successful engagement with the OER agenda.
14. What kind of OERs are being adopted and used?
Again, the evidence in this area can only be limited. OERs in medical education were summarised by one interviewee as having:
Potentially a huge impact, not much yet. ... Good ideas need to be timely, and ideas will resurface when the time is right.
Asked about the type of OERs likely to be most widely adopted and used, this participant's view was typical:
There is interest and appetite in using content that is freely available, especially things like video clips or good quality images, which can be quite difficult to source.
Participants at the e-Learning in Health conference gave examples of the kind of OERs that would meet their own learning and teaching needs:
Videos of healthcare professionals 'to go into the interprofessional working module so they can see e.g. a radiographer talking about what they do'
Video allows students to see what conditions might look like that they might not have come across e.g. a stroke patient walking, that we can't roleplay or mimic.
Simulation is on of the ways that technology has been able to enhance learning as we can't keep practising on patients. We need ethically to have those skills hones before we work on a patient.
In general terms they also wanted e-learning resources to support inter-professional learning and provide insight into different professional points of view, materials that could be referred to 'at the bedside', and good quality simulations giving the opportunity for students to practise procedures without risk to patients.
There was a particular demand for images and video from authentic healthcare settings, and a conviction that e-learning can engage students in novel ways, as well as promoting more creative and reflective teaching.
Given the decreasing investment in resource development in both depts and trusts, there is clearly potential for OERs to be taken up if they are made available to meet these particular learning and teaching needs. Also there is a demand for learning content 'at the bedside' and in other healthcare settings where openly available content may be easier for professionals to access than content held in closed systems.
15. How are OERs being taken up into curriculum processes?
There is certainly more sharing going on in the NHS, mainly informally, but with more formal arrangements e.g. regional content clubs and partnerships coming into existence. Open release remains very uncommon: most examples cited were actually of reciprocal sharing.
Since we set up our image database I have had someone emailing me to say I am using your pictures would you like some of ours.
We have an e-learning regional server that we upload content on, and only we know the web addresses. But we're happy to share those if people want them.
The financial benefits of content clubs and sharing arrangements could be significant if they lead to less outsourcing and more rational use of limited development resources:
so much of it we have to buy off different content providers. I think that has it’s place, but we can do it ourselves, that’s the message that I came away with- that we can do this ourselves
12 months ago where we were almost exclusively using external developers, now we have a collaborative e-learning authoring tool and we're doing much more ourselves.
Student expectations will also have an impact on content development:
As wikivet and virtual patients get more students using them, student pressure will make you make your own stuff better.
However, the potential benefits of OER are currently difficult to appreciate, as there is not yet a critical mass of discoverable, relevant, high-quality content.
I’m somebody who is supposed to know quite a bit about the technology side, within my School, but I still find it difficult to find open resources to know where to go to find the resources we might want.
And the Xpert, I’ve searched, but I haven’t yet found the things that I did want.
I think you have to have X number to start the ball rolling and they weren’t sure how many they could get from across the institution to get that account [iTunesU] open
without them being available people aren’t going to have the expertise to use them and without the expertise they won’t be available, so it’s a chicken and egg situation
It’s about flooding the market... and then you have a choice of stuff, as more stuff is out there you get more choice – supply and demand.
There is still a great need for more good open-source material in my area (biological science).”
16. What have been the impacts on staff?
The main benefits to staff have been a greater awareness of copyright and consent issues, and the consequent likelihood that content is being released and re-used within the law.
Colleagues are starting to ask what the black strip is under the picture – it makes other people think about it as well.
As in other OER development contexts, there are some signs that staff see sharing and open release as a driver for enhanced quality:
[teachers who share material] care about where its going to go and who’s going to look at it... (OER) encourages improvement in quality in teaching.
At QUB, lessons from the PORSCHE workshop are being integrated into a workshop for staff in healthcare which around 50 are expected to attend. At the University of Nottingham, a Veterinary Education symposium in July 2011 promoted the use of Creative Commons, Xerte and Xpert: 'it opened people’s eyes to sharing.' Thanks to a presentation about OLM at a PORSCHE event, London Ambulance Service is working towards a new Oracle Learning Management initiative which should benefit 200 trainers and the 5000 staff who depend on them for training support.
I knew we would get lots from our involvement, which we did. The fact that was done has been incredibly useful for us and we need to go on drive it forward.
There was no evidence of negative impacts on staff beyond the common observations that OER production is intensive of time and attention, and that this is poorly rewarded.
17. What have been the benefits for stakeholders more generally?
18. What has been the overall impact of the project?
The PORSCHE project saw an opportunity to broker collaboration between the separate but closely related cultures of academic medical education and the NHS in its training and professional development role. It was hoped that deeper mutual understanding could develop around sharing of content. The PORSCHE team’s experience of OER development and of working across professions in HE meant that it was in an ideal situation to take this forward. The project did not emerge from the evaluation interviews as having a distinct identity, but was seen as part of the ongoing work of the MEDEV Subject Centre to promote sharing and broker the exchange of ideas in medical education.
You broker the exchange. It makes the whole process easier and faster. You bring together the opportunities. [You] people are key to getting relationships working.
[Thanks to the project, we realised that] we were all doing the same thing, going in the same direction and there wasn’t much sharing that was going on, so it was good to hear about how we can share and how we can collaborate.
There is now better awareness of the legal issues surrounding content and consent, leading in turn to safer practices and reduced risk for individual organisations. Collectively, the NHS is taking a more strategic approach so that policies can be aligned around best practice, and so that sharing of content is recognised as beneficial.
Trusts are also becoming more likely to develop and share content and less likely to buy content in. This cannot be ascribed directly to the PORSCHE project, but the project has definitely provided additional impetus:
The issues [of content development] are now being raised at a nationally level rather than just locally.
The conference definitely tabled the item to discuss a national forum for Ambulance Trusts, and how do we pool our resources together to develop content which is bespoke and specific to us, and what copyright agreements do we fall under and how do we share that, and what costs are associated for development?
Moves towards technical interoperability of Jorum and the NeLR will benefit users of both resources, i.e. educators, trainers and students across all medical educational settings, clinical and academic. The idea of a consent commons, while at an early stage of feasibility, provides another crucial piece of the shared content puzzle. Both developments had yet to bear fruit for individual stakeholders within the timeframe of this evaluation, but the participants who had been made aware of them were excited about the opportunities:
Although the PORSCHE project started off as a technical project in as much as we were developing an interface between the NHS e-learning repository and other open repositories, I think the long-term benefits are more of a cultural change than a technical one.
It is still early days, but people are now talking and thinking about it.
Conclusions
The PORSCHE project has been a productive learning process for the MEDEV subject centre, the NHS e-Learning Repository and Jorum, for partners the London Deanery, Yorkshire and Humber Ambulance Service and members of the Executive Board. On the evidence of this evaluation, it has offered important insights to those touched by its outreach activities, which have in many cases been translated into policy, practice and staff development enhancements in participants' own organisations. On issues of copyright and consent, the project has built effectively on the work of OOER,and established a clear route for individuals and organisations to progress from ignorance through concern to reassurance. From the perspective of this evaluation, everyone who has interacted with the project has felt that engagement to be worthwhile.
From a policy perspective the project has enabled rapid and significant change by providing highly usable guidance, toolkits and exemplar statements, and by aligning these with the needs and existing policy structures of the NHS. A rigorous approach to evidence and evaluation, as demonstrated by the project's comprehensive review of (available) NHS Trust policies, has clearly helped their case.
The project has highlighted significant differences in culture, practice, infrastructure, business case and expertise between academic and clinical stakeholders, despite their shared interest in building quality resources for medical education. At present this manifests in different rates of change towards a situation of more open sharing of content. There is a risk, however, with further funding constraint and a strengthening of the internal market in both HE and the NHS, that the direction of travel could be reversed in either or both sectors. Despite this longer-term uncertainty, in the medium term there is a strong likelihood that the innovations in policy and practice set in train by the project will be sustained, thanks to the focus on building partnerships. The project has also made good progress with adapting messages and resources for a clinical audience, though more needs to be done if the Subject Centre aspires to represent this constituency in parity with academic educators.
In summary, there where were two clear demands from participants in this evaluation study. First, easy-to-use tools to support content release (development, tagging, uploading, syndicating) and re-use (discovery, download and re-purposing/re-branding). Second, for strategic managers to remove some of the systemic barriers to open sharing of content. If the business case for open content fails at the level of individual Trusts and departments, it will require concerted action at a very senior level to recognise and then realise the value of open content to the system as a whole.
In addition PORSCHE was mapped against the programme evaluation criteria.
