FAQ - Is there any evidence that computer-based instruction works?

Answer

There are so many different types of computer-based learning materials that it is almost impossible to give a simple 'yes or no' answer to this question and the debate rages on. Instead, let's break it down a bit into some different kinds of materials and look at some selected research.

Web-based hypertext resources: The majority of these types of resources currently are on-line textbooks, more or less. Some have a great deal of useful information including images, animations, and videos. Do these facilitate learning? Does a textbook? The answer for both is, of course, yes if the person reading the material pays attention and engages at a deep level. A great deal of research has shown that people who read material actively, ie. asking questions as they go, stopping to summarise points, taking notes, anticipating what the author will write next, can learn effectively from on-line or written texts (Kintsch, 1995). However, such learning depends as well on the quality of the resources. Just as some books are not very good, so are some Web pages not very useful. What is _not_ effective is to put up a large amount of poorly structured, fragmented material with no guidance about what is intended to be learned or how to go about it (McKendree et al, 1995). Studies of large hypertexts show that reading from these sorts of materials is poorer than from a well-structured book with the same information and that readers tend not to see all the available information - they get lost in hyperspace (Charney, 1994).

Many studies have shown that adult learners, including university undergraduates, learning from (decently designed) on-line instruction do just as well as those using paper-based media and those learning at a distance (who don't drop out) learn just as well as those in the classroom (given that they spend equivalent time and effort). These 'no significant difference' studies (Clark, 2002) don't really tell us much, though, about what the critical factors are for creating a good or bad resource, simply that students are equally good (or bad) at learning, despite the delivery medium, depending on the quality of the material provided. Therefore, other evaluative criteria might be needed to decide on delivery method for what is basically reading materials - does it allow more flexibility? more opportunity for revision? is it more cost-effective? is it more easily accessible?

Exercise-based software: Some software, particularly CD-ROM, but increasingly also Web-based, have interactive, repeated exercises of some sort. The student must practice some material, usually until a certain criterion is met, the fundamental idea behind Skinner's Teaching Machines and currently in Integrated Learning Systems. A very typical example would be software to let children practice simple addition and subtraction. This can range from drill-and-practice of very specific micro-skills to complex exercises involving multiple steps and deep reasoning.

As summarised in an ERIC report, “Using technology for practice of basic skills can be highly effective according to a large body of data and a long history of use (Kulik, 1994). Students usually learn more, and learn more rapidly, in courses that use computer assisted instruction (CAI). This has been shown to be the case across all subject areas, from preschool to higher education, and in both regular and special education classes…Fletcher, et al (1990) reports that in the military, where emphasis is on short and efficient training time, the use of CAI can cut training time by one third. In the military, CAI can also be more cost-effective than additional tutoring, reduced class size, or increased instruction time to attain equivalent educational gains.” (Kosakowski, 1998).

This kind of software can indeed lead to better mastery of the types of procedures that lend themselves to repeated practice, but research has shown that spaced exposure is very important - 10 minutes per day 3 or 4 times a week is more effective than 30-40 minutes at one go (Underwood et al, 1996, 1997). This is a result backed up by decades of psychological research on learning in general (and which dictates against cramming for a single exam, too!).

And again, the exercises need to be well-designed. The crucial issue of instructional design is important in all forms of teaching, of course, and is one that is too frequently ignored (Merrill, 1997). Another possible reason for effective learning in these systems is the delivery of instantaneous feedback. Feedback is critical for learning, especially when skills are involved (McKendree, 1990). Another reason for improved learning over, for instance, paper-based exercises is that students often spend more time working on computer-based exercises and finish more examples, both of which lead to better learning.

Simulations: A giant step up in complexity from these practical exercise systems is simulation. Here, a student must work through an exercise of some sort, but one that exemplifies some of the complexity of a real-world situation. Simulations may involve instructional feedback or may not; they may be highly complex or relatively simple. These types of programmes can lead to very effective learning, if they are capturing the important characteristics of the situation the student is intended to be learning. It is fairly obvious that a high-fidelity (and highly expensive) simulator such as those used for teaching airplane pilots or surgeons can be effective to a high degree. There is also some evidence that simulation-based learning is retained longer than that from classroom instruction.

However, looking at less high-fidelity software, there is sometimes a problem with students becoming proficient at playing the 'game' without learning the underlying lessons. An example is SimLife in which students could run the simulations, but learned nothing about ecology, which was the intended result (Turkle, 1995). It is essential to combine unstructured simulations such as this with discussions of the concepts they are intended to be illustrating and specific tasks or assessments to work on. Even better is to make sure proper structure is built into the simulation to encourage learning – back to instructional design again. An example of a software based simulation that can work well is BGuILE which simulates the activities, in a very structured environment, of collecting data and analysing it to learn about variation and microevolution in finches from the Galapagos Islands. However, even with this sophisticated software, research showed that it was important for students to discuss the underlying ideas in regular debriefing sessions, as well as to accomplish the specified tasks, to bring about optimal learning (Reiser, et al., 2001). Any concrete exercise, computer-based or not, benefits from being explicitly framed in the larger domain concepts (McKendree, 2002).

Cognitive tutors: Tutoring systems based on cognitive models of the learning that students are intended to achieve can be extremely effective. Studies of the ACT-R tutors for algebra, geometry, and programming have found that students learn material in one-half to one-third the time needed in traditional classrooms and perform better on standardised tests. It also seems to benefit the weaker students the most (Anderson, et al., 1995 ; Koedinger et al, 1997). Again, these results are probably due to a combination of factors: extended practice, instant feedback, very strong instructional design based on detailed models of learning, and supporting materials (including the teacher) to enhance conceptual learning. However, these tutors have only been designed for very procedural tasks - areas in which there are problems that can be set and worked through in a fairly algorithmic way - and are expensive and time-consuming to build, so can only really be created for domains in which curriculum goals are not very likely to change rapidly.

Summary:

• Computer-based learning can be effective or ineffective, depending highly on quality of instructional design and implementation
• Unless students are extremely motivated, very few computer-based resources are effective for learning unaccompanied by any supporting teaching or materials, just as it is quite difficult to learn complex domains by only reading a book
• Also, of course, many additional factors influence success such as whether the use of resources is required or not, proper administrative and technical support, adequate teacher training and student support, and adequate financing for design and maintenence

References

Anderson, J. R., Corbett, A. T., Koedinger, K., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. The Journal of Learning Sciences, 4, 167-207. Available from: http://act-r.psy.cmu.edu/papers/Lessons_Learned-abs.html

Charney, D. (1994). The Impact of Hypertext on Processes of Reading and Writing. In Hillgoss, S. and Selfe, C. (Eds) Literacy and Computers, New York: Modern Language Association, pp. 238-263.

Clark, R. (2002) The No Significant Difference Phenomenon, retrieved Dec. 2002, available at: http://teleeducation.nb.ca/nosignificantdifference/

Fletcher, J.D., Hawley, D.E., & Piele, P.K. (1990). Costs, effects, and utility of microcomputer assisted instruction in the classroom, American Educational Research Journal, 27, 783-806.

Kintsch, W. (1998). Comprehension: a paradigm for cognition. Cambridge: Cambridge University Press.

Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8, 30-43.

Kosakowski, J. (1998) ERIC Digest ED420302, Downloaded 09/12/02, http://www.ed.gov/databases/ERIC_Digests/ed420302.html

Kulik, J.A. (1994). Meta-analytic studies of findings on computer-based instruction. In E.L. Baker and H.F. O'Neil, Jr. (Eds.), ""Technology assessment in education and training."" Hillsdale, NJ: Lawrence Erlbaum.

McKendree, J. (1990). Effective feedback content for tutoring complex skills, Human-Computer Interaction, 5, pp. 381-413.

McKendree, J., Reader, W. and Hammond, N. (1995). The homeopathic fallacy in learning from hypertext, Interactions, Vol 2, No, 3, pp. 74-82.

McKendree, W. (2002). The role of discussion in learning, Poster presented at AMEE 2002. Available from: http://www.ltsn-01.ac.uk/ltsn_images/pdfs/mcKendree_amee_2002.pdf

Merrill, D. (1997). Instructional strategies that teach, CBT Solutions, Nov/Dec, 1-11. Available at: http://www.id2.usu.edu/Papers/Consistency.PDF (retrieved Dec 2002).

Reiser, B. J., Tabak, I., Sandoval, W. A., Smith, B., Steinmuller, F. and Leone, T. J. (2001) BGuILE: Strategic and Conceptual Scaffolds for Scientific Inquiry in Biology Classrooms, In S.M. Carver & D. Klahr (Eds.) (2001). Cognition and Instruction: Twenty five years of progress.

Mahvah, NJ: Erlbaum. This and other BGuILE Papers available from: http://www.letus.org/bguile/Papers/Bguile_papers.html

Turkle, S. (1995). Life on the Screen: Identity in the Age of the Internet. Phoenix Press. Underwood, J. & Brown, J. (eds.) (1997) Integrated Learning Systems: Potential into Practice. Heinemann

Underwood, J., Cavendish, S., Dowling, S., Fogelman, K. & T. Lawson (1996) Are integrated learning systems effective learning support tools? Computers & Education, 26 (1-3), 33-40.

Disclaimer: This FAQ was written by Dr Jean McKendree and does not reflect an official endorsement by the HEA or any other organisation.  Any questions or queries should be sent to: enquiries@medev.ac.uk

Last updated: 01 July 2011