MEDEV

School of Medical Sciences Education Development

PORSCHE survey quantitative analysis

There were 10 questions in the survey in total, and what follows is a quantitative analysis of responses not reported in the free text analysis. In all bar charts, numbers refer to the percentage of responses. The survey was completed by a total of 134 recipients. A quantitative analysis of responses is reported below using the in-built online SurveyMonkey system. In all bar charts, numbers refer to the percentage of responses.

Question 1. What best describes you?

This question confirmed our target audience. Survey respondents represented both the HE (31%) and NHS (26%) domains, including academic (40%) and healthcare sectors (11%). There were respondents with roles such as e-learning technologists (27%), librarians (17%), researcher (12%), senior management (13%) and clinicians (8%). Medicine, dentistry and veterinary medicine disciplines were all represented. Survey respondents were largely from the UK, although 6% were not from the UK and included Ireland, Europe and the US.

Question 2. Have you attended any of these PORSCHE-related dissemination events?

It was our intention to recruit both PORSCHE event attendees (workshops, conferences, engagement meetings) and a comparative control group of non-event attendees. Survey respondents were recruited from all our events, with greater a proportion being derived from our more numerous pools of conferences delegate and one of the larger workshops as would be expected. The control group was invited from both MEDEV and the NeLR. The relative proportion of both attendees and non-attendees were similar (55% and 45% respectively).

Figure 8. PORSCHE survey question 2.

Question 3. Which words best describe this event? (Kirkpatrick evaluation level 1, ‘reaction’)

These were parallel questions for event attendees and the control group. Survey respondent were directed to either question three or seven depending on survey pathway logic relating to event attendance or not. For event attendees this was designed to record reaction to the content of an event (Kirkpatrick evaluation level 1, ‘reaction’). The control group who could not give their reaction to an event were instead presented with five statements relating to open content sharing principles derived from the PORSCHE workshop intended learning outcomes:

Good practice in creating and sharing learning resources requires:

• understanding the difference between copyright ownership and licensing
• content creators and third party content needs to be adequately attributed
• learning resources need to clearly indicate the copyright status of any works
• a Creative Commons open licence should be used
• informed consent needs to be documented for resources containing patient data and recording of other people

Following these steps allows you to manage risks and safely share content.

Although not ideal, the team hoped this approach for the control group would collect at some broadly useful comparative data.

‘Reaction’ – control group versus attendees

Figure 9. PORSCHE survey question 7 (control group, Kirkpatrick level 1 ‘reaction’). 

The reaction of the control group was on balance positive with the highest positive survey question option of ‘relevant’ being reported at 47%. However, this question highlighted a distinct smaller sub-population of non-event attendees who had a negative reaction to the intended learning outcomes of the PORSCHE workshop. It can be inferred that this sub-population represents members of the constituency who do not have a positive reaction to the concept of open content sharing and would be unlikely to attend one of our workshops by choice. Its well known that there are people not engaged by the ‘OER movement’ for reasons such as it they believe it is a passing trend, is overly philanthropic, or is not sustainable. Some of these people can be successfully lobbied, others not. There is no doubt that outside of funded projects there is still plenty of engagement work to be done.

Figure 10. PORSCHE survey question 3 (event attendees, Kirkpatrick level 1 ‘reaction’). 

The reaction of the event attendees to the content of the events was largely positive with the highest positive survey option of ‘informative’ being reported at 59%. A small number negative option responses were received (‘baffling’, 3% and ‘irrelevant’, 2%), but noticeably less than the control group.

Question 8. I am confident in ...? (Kirkpatrick evaluation level 2, ‘learning’)

Survey respondents were asked to indicate (‘yes’, ‘no’, ‘neutral’ or ‘not applicable’) if they were confident in options related to open content sharing practices derived from the content of PORSCHE workshops. The aim of this question was to measure Kirkpatrick evaluation level 2 (‘learning’).The use of the term ‘confident’ was considered, on balance, to be an appropriate although a high standard to attain. If event attendees were not confident in aspects of OER good practice then it was unlikely they would share open educational resources in the future.

‘Learning’ – control group versus attendees

Figure 11. PORSCHE survey question 8 (control, Kirkpatrick level 2, ‘learning’).

Figure 12. PORSCHE survey question 9 (attendees, Kirkpatrick level 2, ‘learning’).

The event attendees had greater confidence in many of the survey options reflecting open content sharing good practice. This included where to find openly licenced learning resources (control 51% ‘yes’ versus event attendees 63%), using Creative Commons licenced resources (64% versus 75%), availability of good practice toolkits (32% versus 48%) and sharing resources with students/colleagues (30% versus 39%).

In addition, event attendees the proportion of ‘neutral’ response compare to ‘no’ response across many of the options was greater. This may be a positive effect, showing transition from ‘no’ confidence to ‘neutral’, and requiring further time and practical application of event attendee knowledge to gain sufficient confidence.

Some of the results were perhaps paradoxical in that confidence in some options was reduced. This may represent false confidence in the control groups assessment of their knowledge in some areas. For example, in the survey option investigating copyright and licencing differences, confidence was less in event attendees (control ‘yes’ 65% versus 59% attendees). This discrepancy may also be alluded to in respect to the ‘no’ responses to this question where the control group was 30% compared to 15% for event attendees. As assessment of learning took place through self-assessment of confidence rather than through formal assessment, this was a limitation to the survey analysis, but necessary because of the practicalities of the project.

Question 9. I have personally ...? (Kirkpatrick evaluation level 3, ‘behaviour’)

The aim of this question was to investigate to what degree PORSCHE event attendees has taken their learning to date in respect to open content sharing principles and actually implemented it into their current work and practice. This represents Kirkpatrick evaluation level 3 ‘behaviour’. The survey options were similar to above, but respondents were ask to report what they had actually done in practice. In addition, respondents were asked about if they had disseminated any of their knowledge on open content sharing to colleagues (‘behavioural dissemination’).

‘Behaviour’ - control versus attendees

Figure 13. PORSCHE survey question 9 (control, Kirkpatrick level 3 ‘behaviour’).

Figure 14. PORSCHE survey question 9 (attendees, Kirkpatrick level 3, ‘behaviour’).

The event attendees had greater adoption into their behavioural practice of several of the open content sharing principles compared to the control group. This included searching for open content (control 65% versus 75% attendees), considering liability (control 54% versus 60% attendees) and disseminated good practice (control 43% versus 51% attendees). On balance there would appear to be a rate limiting step between learning about open content sharing practices, to the point of confidence and when this knowledge/confidence can be put into practice. Also, to put some aspects of knowledge of good practices to actual use required permission from individuals’ institution and which is often a barrier.

While the increase in the choice of the adopted best practices in digital professionalism option (control 43% versus 51% attendees) could be expected, this may in part result from a greater familiarity with the terminology. However, the trend is consistent with some of the individual components of digital professionalism.

There is a notable disparity between the reported consideration of risk when creating learning resources and the use of a good practice toolkit. This is demonstrated in both control and event attendee groups (control considered 54% versus 18% control used; attendees 60% considered versus 19% attendees used). Further investigation may ascertain if this was because individuals believed they had the sufficient knowledge needed to create content without risk (and this could be verified). Alternatively, this may indicate further promotion of the various good practice toolkits is need and/or there are barriers to their use (i.e. perceived time demands, usability). Overtime (and individuals experience with open content sharing) toolkit usage may drop as good practice knowledge is adopted and becomes a routine part of practice, however research to suggests this point has not been reached yet for the population as a whole.

As discussed previously some of the survey options give apparent paradoxical results (at least that what would expect). For example, reporting of dealing with consent issues (control 43% versus 27% attendees) is reduced in event attendees. Similarly, reporting of correctly attributing content creators is reduced in event attendees (control 71% versus 58% attendees). Based on practical experience within MEDEV constituencies, including in-session feedback/comments during workshops and as reported in the OOER case studies,  the hypothesis that overall demonstration of these abilities with the population is not at this level, can be maintained. Event attendees may be more critically aware of their own past activities and can assess their own behaviour practice more accurately. To express this theory in a different way, this may represent the concept of ‘you don’t know what you don’t know’. Certainly it would possible to test this concept in the future through assessment.

Question 10. My employer/institution ...? (Kirkpatrick evaluation level 4, ‘results’)

This question set out to explore if there was any measurable correlation in good practice adoption (arising from behavioural change of individuals who attended a PORSCHE project event) and any institutional changes that this resulted in. This represented the modified Kirkpatrick evaluation level four (‘results’), which is the highest level and the most difficult to achieve. Certainly it would be ambitious to detect measurable formal institutional change (and which had already impacted on the individual responding) within the timescales of the project, however, the OOER project had previously reported resultant institutional policy changes. The survey would at least provide baseline data for potential future comparison. Thus, the aim of this question was to assess the degree of institutional support, training, reward/recognition and barriers to open content sharing practice experienced by our survey respondents. These could be represented through formal or informal institutional policies. The results were used to assess if and how level 3 ‘behaviour’ and level 4 ‘results’ were connected, for example, did individual’s good practice adoption have a ‘bottom-up’ effect (i.e. ‘result’) on their institution as we hoped? Conversely, it would be interesting to note if OER supportive institutions (if sampled by survey) demonstrated a ‘top-down’ effect on their employees’ behaviour?

 ‘Results’ – control versus attendees

Figure 15. PORSCHE survey question 10 (control, Kirkpatrick evaluation level 4, ‘results’).

 Figure 16. PORSCHE survey question 10 (attendees, Kirkpatrick evaluation level 4, ‘results’).

The control group reported 49% institutional support for use of openly licenced resources. This could  indicate that engagement with open content sharing and practices at an institutional level needs continued promotion. Moreover the response from event attendees showed a reduced reporting of this support (49% control versus 40% attended). This may be due to the fact that event attendees (especially workshop attendees) choose to attend events because they need more information and support on these issues. Alternatively, it may be as previously proposed that post event attendees are more informed and constructively critical of their institutional policies and support.

Event attendees reported greater value of the use openly licensed resources in their institutions (29% control versus 36% attended).This may be due to the fact that event attendees (especially workshop attendees), by the nature of them attending related events, are receiving more institutional information and support on these issues. Alternatively, it would be rewarding to know if event attendees post event activities (such as ‘behavioural dissemination’) were effecting this change, however this would required further study to elucidate. A similar trend was seen for the option ‘sees value in sharing’ (17% control versus 27% attended).

Again, there was initially paradoxical increased response proportions in event attendees (compared to control group) for ‘neutral’ response choices for institutional training to use openly licensed resources (attended, 17% ‘neutral’; control, 10% ‘neutral’) and reward to share openly licensed resources (attended, 23% ‘neutral’; ) training to use openly licensed resources (control, 10% ‘neutral’; control, 10% ‘neutral’). Thus, event attendees appeared to be more critical of their institutional reward and training with regards to open content sharing. This could act as a driver for them to instigate the process of institutional policy change on these matters.

As a trend, the proportion of ‘don’t know’ responses for all questions options in this section was reduced compared to control group (30% control versus 22% attended). Thus event attendees are more informed on what is and is not occurring (support, training, rewards, barriers, etc.) in their respective institutions with regards to open content sharing. Also as a trend, there were reduced positive responses generally for all question option related to ‘sharing’ of open content versus ‘using’ open content at an institutional level. There appeared to be an imbalance between institutional support in using and sharing content and for open content sharing to be sustainable this may need to be addressed with more bi-directional (circular) practices.

Numerical sum scoring of Kirkpatrick evaluation level attainment

The question option responses from individual survey respondents were scored and summed for each of the modified Kirkpatrick levels. Attainment in the control group and event attendees (conference and workshops) against the evaluation levels of reaction (negative and positive), learning, behaviour and results was displayed on the box plot below. In the control group there was a smaller positive numerical score for the Kirkpatrick level 2 (learning) compared to conference and workshop attendees (control, 1.5; conference, 1.8; workshop, 5.4). In workshop attendees the greatest level ('peak') in Kirkpatrick level attainment was at learning. It is expected to be more difficult to progress from the lower to the upper levels of the Kirkpatrick scale, so this was not surprising. For workshop attendees, time to incorporate learning into behaviour and institutional support and policies allowing them to fully adopt open content sharing good practice is needed. It would be useful to assess the respondents in the future to analyse this. Conference events increase overall learning to some extent and perhaps some of the behaviours associated with good practice. However, workshops (i.e. significant time and dedicated staff training) alone seem to be truly effective in changing overall behaviours of attendees towards adoption of a full set of open content sharing good practice characteristics, as assessed by the numerical scoring system.