Evaluating the Effectiveness of e-Learning

Resources need to make evaluation work

"Much technology is available to support evaluation execution and reporting needs. The evaluator needs to research and select from the many vendors and programs available, pinpointing the most appropriate technology for the organization. Hardware and software can support almost any phase of the evaluation, including data collection, data management and reporting, and data analysis." Jack J. Phillips, Patricia Pulliam Phillips, and Toni Krucky Hodges, Learning Circuits, November 2004

Added: 9 November 2004
Reviewer's Note:
This is an excerpt from the new ASTD Press book,Make Training Evaluation Work (http://store.astd.org/product.asp?prodid=2849&deptid=)

Showing the value of e-learning

"Providing instruction across the Internet and other computer networks is a relatively new phenomenon. To some extent, we are just learning how to harness this media for learning, and stakeholders are rightly asking whether our efforts are paying off. Establishing value for any venture is, at its most basic level, about determining what goals are important, selecting measures to show if these goals are achieved, and then measuring and analyzing. This is true for e-Learning, too." Patti Shank, eLearningGuild, December 2003

Added: 4 April 2004
Reviewer's Note:
Results of a survey run by eLearning Guild October-December 2003

Evaluating the effectiveness of e-learning strategies for small and medium enterprises

"The aim of this paper is to provide a framework to measure the effectiveness of e-learning strategies or programmes. This will be undertaken without reference to any particular e-learning programme or strategy." Eduardo Figueira, February 2003

Added: 18 March 2004
Reviewer's Note:
A paper from European seminar: exploring models and partnerships for eLearning in SMES

Is Kirkpatrick obsolete? Alternatives for measuring e-learning success

"In recent years there has been growing criticism of Kirkpatrick's approach to evaluating training programs. Kevin Kruse (the e-learning guru) points out "critics of the Kirkpatrick model say that it doesn't take the business impact far enough and that the final step in any training program should be a "fifth level" of evaluation -- financial return. This ultimate evaluation determines the financial return on investment (ROI) of the training program"." Kaliym Islam, Learning & Training Innovations Magazine, 18 February 2004

Added: 22 February 2004
Reviewer's Note:
Kaliym presents the case for Six Sigma

A New Methodology for Evaluation: The Pedagogical Rating of Online Courses

"Online courses should be evaluated not merely for their online materials, but for the sum total of elements contributing to their success. In this article, Nish Sonwalkar proposes an evaluation tool to provide an overall measure of the effectiveness of an online course-with consideration given to the inclusion of various learning styles, media types, and interactive elements." Dr Nishikant Sonwalker, Syllabus Magazine, January 2002

Added: 31 January 2004
Reviewer's Note:
"The pedagogy effectiveness index and the summative evaluation instrument used in combination can be powerful tools for evaluating large numbers of online offerings."

Criteria for evaluating the quality of online courses

"The criteria presented in this guide are based on the national and international experiences of staff in the Instructional Media and Design department at Grant MacEwan College. Although they were developed to assist educators in evaluating the effectiveness of online courses, they may also be used as guidelines for course developers." Clayton R Wright, Grant MacEwan College

Added: 9 December 2003
Reviewer's Note:
Also available in MS Word format to download

Learning measurement: it's not how much you train but how well

"A key consideration in designing e-Learning measures is deciding what to measure and report. Measurement can be expensive and time-consuming. This has strategic implications for budgeting as well as for continuous improvement of e-Learning. Read this article to identify not only the measures, but also a system for applying them in a way that does not break the bank!" Jeffrey Berk, eLearning Developers Journal, 3 November 2003

Added: 5 November 2003
Reviewer's Note:
You will need to subscribe to the eLearning Guild to access this article

Quality and effectiveness of e-learning

"How do e-learning practitioners rate the quality and effectiveness of their e-learning programs? Where are the soft spots among e-learning technologies used by practitioners? How do perceptions of quality and effectiveness of e-learning vary by region?" Tom Barron, Learning Circuits, May 2003

Added: 25 May 2003
Reviewer's Note:
"The 24-question, Web-based survey, conducted with the participation of ASTD and other LoD partners, garnered nearly 350 responses from practitioners around the globe."

Getting bigger slices of the e-learning pie

"The e-learning pie is a simple tool that can help you assess the success of your e-learning projects." Brooke Broadbent, Learning Circuits, May 2003

Added: 25 May 2003
Reviewer's Note:
8 criteria for assessment and steps to assess the state of e-learning projects

The web's impact on student learning

"A Review of Recent Research Reveals Three Areas That Can Enlighten Current Online Learning Practice." Katrina A Meyer, THE Journal

Added: 15 May 2003
Reviewer's Note:
Also flags up where research still needs to be done in this area

Measures of learning effectiveness

"Although many CEOs value training because they believe it strengthens the organization and serves as a retention tool, not many are clear on how to measure the return on the investment (ROI). With increased pressure to justify expenses, CLOs are looking for ways to show improved bottom-line results." Michael Brannick, Chief Learning Officer Magazine, January 2003

Added: 10 January 2003
Reviewer's Note:
"Consider that the complete integration of testing and assessment tools and programs can give you the data and feedback you need to prove not only that your program is meeting its objectives of training employees effectively, but also that the organization is receiving some real, tangible benefits because of its existence."

Quality auditing of online learning

"Guidelines and Professional Development Resources for Auditors of Online Training Programs" flexiblelearning.net.au

Added: 31 October 2002
Reviewer's Note:
"Updated version of the draftguidelines incorporating feedback from the consultations."

An Assessment of the Effectiveness of e-learning in Corporate Training Programs

"Corporate managers are constantly looking for more cost-effective ways to deliver training to their employees. E-learning is less expensive than traditional classroom instruction. In addition, many expenses - booking training facilities, travel costs for employees or trainers, plus employee time away from the job - are greatly reduced. However, some firms that have spent large amounts of money on new e-learning efforts have not received the desired economic advantages." Judith B Strother, Florida Institute of Technology, International Review of Research in Open and Distance Learning, April 2002

Added: 27 July 2002
Reviewer's Note:
Conclusion: "While few people debate the obvious advantages of e-learning, systematic research is needed to confirm that learners are actually acquiring and using the skills that are being taught online, and that e-learning is the best way to achieve the outcomes in a corporate environment. This research must be grounded in solid theoretical precepts to assure that meaningful results are obtained."

Final Analysis

"How AT&T adapted Kirkpatrick's evaluation tools to e-learning then applied the same rigor to selecting vendors." Michael Welber, e-learning Magazine, 1 June 2002

Added: 7 June 2002
Reviewer's Note:
"The group discovered some surprising results as they put both the training and the vendors through the paces."

Is digital learning effective in the workplace

"One should probably expect the title of this paper to be a statement instead of a question. However, the question is one that arises with ever increasing frequency in both corporations and academic instituitions throughout the world. Hence, it is a question that deserves more attention than a recounting of the plethora of anecdotal accounts and projections from technology and business analysts." Larry G Moyer, eLearn Magazine

Added: 22 May 2002
Reviewer's Note:
A 6-page article with research and conclusions

Course evaluator

"We have created a course evaluator tool based on our 96 OID Standards. To use this tool, you will need to have a copy of Microsoft® Excel® and must be able to read Adobe® Acrobat® files." Michigan Virtual University

Added: 20 March 2002
Reviewer's Note:
Based on the open standards devised by MVU you can evaluate your own course

Measuring e-learning's effectiveness

"In this article I review a proven methodology for measuring effectiveness, which is the first part of really measuring return on investment (ROI). Over the years of deploying large e-learning programs at DigitalThink and other companies, I found that this process works flawlessly, and is easy to understand." Josh Bersin, e-learning magazine, March 2002

Added: 13 March 2002
Reviewer's Note:
Provides some useful ideas to consider how you evaluate your e-learning programs

Measuring success: Evaluation strategies for distance education

"Only by evaluating the effectiveness of DE programs can we justify their use and continue to develop their quality." Educause Quarterly, 2002

Added: 3 March 2002
Reviewer's Note:
Covers stages of formative and summative evaluation

A new methodology for evaluation: the pedagogical rating of online courses

"Online course offerings are increasing in number every day. Most universities and corporate training facilities now offer some or all of their courses online. In fact, more than 1,000 corporate universities and online providers offer courses in everything from information technology to Chinese cooking. Although it is clearly advantageous for asynchronous learners to access educational information and content anywhere and anytime, it is difficult to evaluate the quality and effectiveness of online courses and learning modules" Nishikant Sonwalkar, Syllabus Magazine, January 2002

Added: 15 January 2002
Reviewer's Note:
Nishikant proposes a new instrument for evaluating online courses

What matters in judging distance education? Not how much it's like a distance course

"Barbara B. Lockee, an assistant professor of instructional technology at Virginia Tech, is an instructor in a graduate program on how to teach at a distance. The classes are held both face to face and -- not surprisingly -- at a distance. She says that trying to judge the quality of a distance course by comparing it with a traditional course is misguided." Dan Carnevale, Chronicle of Higher Education, February 2001

Added: 18 July 2001
Reviewer's Note:
An instructor's view of how to evaluate an online course

Seven principles of effective teaching: A practical lens for evaluating online courses

"The "Seven Principles for Good Practice in Undergraduate Education," originally published in the AAHE Bulletin (Chickering & Gamson, 1987), are a popular framework for evaluating teaching in traditional, face-to-face courses. The principles are based on 50 years of higher education research (Chickering & Reisser, 1993). A faculty inventory (Johnson Foundation, "Faculty," 1989) and an institutional inventory (Johnson Foundation, "Institutional," 1989) based on these principles have helped faculty members and higher-education institutions examine and improve their teaching practices." Charles Graham, Kursat Cagiltay, Byuung-Ro Lim, Joni Craner and Thomas M. Duffy, the technology source, March/April 2001

Added: 18 July 2001
Reviewer's Note:
Some good principles to help with identifying good practice

New times demand new ways of learning

"Recent research builds a powerful case against what used to be accepted "truths" about learning and technology. First, there is strong evidence that traditional models of learning, traditional definitions of technology effectiveness, and traditional models of the cost effectiveness of technology don't work." This is one section in the NCREL's project "Plugging In: choosing and using educational technology

Added: 27 June 2001
Reviewer's Note:
A very thorough look at how we should be evaluating the effectiveness of learning in a technology-based learning environment

"No significant difference" phenomenon

"This site provides selected entries from the book "The No Significant Difference Phenomenon" as reported in 355 research reports, summaries and papers - a comprehensive research bibliography on technology for distance education." Thomas Russell

Added: 27 June 2001
Reviewer's Note:
This is a very interesting collection of papers that suggests there is "no significant difference" in learning no matter what kind of media or methods were used.

E-Learning could do better

"Peter Honey describes his two lingering doubts about the effectiveness of e-learning - the first is about the motivation to learn and the second is that e-learning assumes that people already know how to learn." December 2000

Added: May 2001
Reviewer's Note:

Evaluation for distance educators

#4 Distance Education at a Glance, College of Engineering, University of Idaho

Added: 2000
Reviewer's Note:

Evaluating Learning in Virtual Environments

"The project is itself innovative in the way that it integrates three key functions--research on learning and development, classroom practice, and authentic assessment of both individuals and programs--in learning environments made possible via new technologies." (1997) University of Texas.

Added: 2000
Reviewer's Note:

Evaluation: a practical guide to methods

This article, from the Implementing Learning Technology series, "seeks to cover the principles for conducting an evaluation whether it is a small or a large project. An understanding of the theory and background to evaluation is beneficial in order to better plan, design and conduct an evaluation programme. Above all there is no substitute for having a clear purpose for an evaluation, defining the right question is a prerequisite. (1997) LTDI (Learning Technology Dissemination Initiative)

Added: 2000
Reviewer's Note:

Evaluation Cookbook

"A practical guide to evaluation methods for lecturers." (1999) LTDI.

Added: 2000
Reviewer's Note:

Observing, Measuring or Evaluating Courseware

"Numerous people are involved in some way in introducing learning technology into teaching, whether in acquiring and using some software developed elsewhere or in authoring new software. Having put in considerable effort during a project, we generally wish (or are required by others) to be able to show something about the results. Simply delivering the software on a disk is seldom felt to be enough: what can we do to pull together and present further evidence?" Steve Draper, Dept of Psychology, Glasgow University

Added: 2000
Reviewer's Note:

The case for Level 3

"How do you determine the most relevant and effective measures and evaluation methods for e-learning? According to new research, focusing on Level 3--job performance-based measures--may be the most promising strategy." By Brandon Hall and Jacques LeCavalier, Learning Circuits, November 2000

Added: 2000
Reviewer's Note: