Evaluation of Three Educational
Online Delivery Approaches
Clark Edwards, Ph.D.
Janie Harden Fritz, Ph.D.
Introduction
A discussion of the potential of alternative education delivery technologies
usually ends with the question: "Is learning as effective when delivered
via alternative media as opposed to that which happens in the face-to-face
classroom environment?" This is a reasonable question, given the costs
associated with production of much media-based learning materials and technologies.
The operative word in that question is"effective," and more accurately
refers to the effectiveness of "learning" rather than "teaching."
As Illich opined in the 1960s, that with technological resources education
could become learning rather than teaching. The "resources" Illich
wrote of in the 1960s is very much like what the Internet has become and
is becoming in higher education. The Internet, as Sangster (1996) has noted,
is about learning not teaching. "Learning how to learn is all about
being able to adapt...being able to take past experiences and use them in
the future, both in and out of the original context, being able to take
a skill and alter or adjust it to work in a different context, and being
able to learn new things when required." In 1983, Richard Clark reviewed
the research to that date on media-delivered education and disturbed many
with his conclusion that "media are mere vehicles that deliver instruction,
but do not influence student achievement any more than the truck that delivers
our groceries causes changes in our nutrition" (Clark, 1983).
The fundamental questions regarding the learning "effect" seem
little resolved after a decade. However, what is becoming more clear is
the mediation process places greater responsibility on the "learner"
side of the equation. On one side of the debate are those who contend that
each medium has a unique set of characteristics that differentially support
different types of learning activities, goals and outcomes. Supporters of
this view argue that "understanding the ways in which students use
the unique processing capabilities of the computer [or other media] is essential
to understanding the influence the computer may have on learning and to
building media theory" (Kozma, 1994). The students' perceptions about
the use of three styles of mediated delivery is a significant focus of this
study.
Clark, on the other side of the debate, continues to argue that "media
will never influence learning" (Clark, 1994) and that differences in
instructional design and learning activities, applicable to almost any medium,
are the causes of differential learning outcomes under different media treatments.
At the root of the debate is the complexity of the learning process itself.
Certainly, one facet of this view must include the students' self-evaluation
of the learning process. In fact, if a student believes they have learned,
that should be the starting point for all evaluative processes in measuring
learning.
In some contexts, learning means acquiring a set of conceptual facts and
ideas that can be quickly recalled during formal examinations. In others,
learning includes the ability to integrate new knowledge critically or creatively
into unique contexts. Some educators include affective responses such as
"appreciation and enjoyment" in a full definition of learning.
Some argue that learning has not occurred at all unless it can be demonstrated
behaviorally. O'Donnell suggests that in addition to the problem of definition,
there's another facet to consider:
Every revolution in communication has both added to the power and range
of what is communicated, and taken away some of the intimacy. Writing began
the long, slow disestablishment of the face-to-face community of people
who all knew each other, and every communication technique introduced since
then has furthered that
process.(O'Donnell, 1995)
Despite the intricacies of the debate, research over the past 70 years has
generally concluded that there are no significant differences between learning
delivered face-to-face and that delivered by alternative media. This generalized
result is used by media proponents to argue for the advantage of media delivery,
since no direct human intervention is involved in the learning outcomes.
Traditional delivery proponents have argued that since no learning improvements
were noted, there is no compelling reason to change from time-proven (face-to-face)
delivery methods.
In this study, thee distinct approaches have been taken which embody three
levels of the traditional approach and three differing levels of the mediated
delivery approach. These approaches were taken with given that research
to date has not shown complete support for either approach alone. Therefore,
a reasonable optimum must be somewhere in the "middle" or somewhere
in combination of the new and the traditional approach. In fact, this initial
research suggests that it's not a question of "either / or" with
regard to the use of mediated materials versus traditional human intervention,
rather the consideration is how much of which facilitates the best learning
situation.
O'Donnell, an experienced user of teaching technologies, argues for the
integration of technology even while educators strive to determine what
"teaching" method or methods are better.
"My experience these last years has been that the new technologies
of networked information are indeed liberating, to real teachers and real
students. It's not as though we couldn't use some help. There are plenty
of frustrations for teachers, plenty of obstacles yet to surmount, plenty
of barriers separating us from the students we want to reach. The best way
to view information technology is to let it address the problems we already
know we have (O'Donnell, 1995)."
The technology of communication, however, is far from neutral, and as McLuhan
points out, is loaded on the "learning" side of the equation:
"When technology extends our senses a new translation of culture occurs
as swiftly as the new technology is interiorized (McLuhan, 1962)."
Addressing the specifics of the Internet, Teilhard de Chardin went further
to predict that a web of communication technology would first grow up beside,
then surround and finally would be organically assimilated into human consciousness
(de Chardin, 1973). With the extraordinary development of the Internet in
higher education, the futures predicted would seem upon the collective academy
now, and, the question of "how" best to implement the learning
experience the most pressing.
Alternative delivery styles
Research to date suggests that quality learning can be delivered in many
ways, by many media, including face-to-face delivery. Given this rough-equivalency
of outcome, we can legitimately move to questions of cost-effectiveness
and access. Face-to-face delivery is expensive and cannot scale itself up
to meet ever increasing demand. Higher education in the United States has
increased in cost over the past 25 years at rates 3-5 percent above those
of inflation (Gillford, 1994). Universities are challenged by the demand
fueled by larger percentages of high school graduates attending university,
the need for life-long learning and retraining, and the increase in technological
skill levels demanded by information-age employment.
The most compelling argument for the alternative delivery of university
programming comes from the consumers themselves -- the new students of the
21st Century. Students, as consumers, are demanding and are willing to pay
for education and learning experiences which can be delivered at the time
and place where that learning is most relevant and convenient. It is easy
to imagine this type of learning in content areas such as engineering where
knowledge of emerging technologies can have direct job application. But
there are also opportunities for philosophers, historians and others in
the humanities and social services where events such as personal crisis,
changing vocations, or personal interest create the need for a learned response.
The response to the challenge of alternative media delivery systems must
be a willingness and desire to explore many new options. There is little
to guide educators as to the better methods to employ to produce the greatest
pedagogical support. In a 1966 study of 660 institutions, only 43% had a
strategic plan in place to cope with the goals, objectives or implementation
for the role of information technology in higher education (Green, 1996).
Green also noted that the national survey of college campuses revealed that
"More than a decade into what some have called the 'computer revolution'
in higher education, it is very clear that most campuses are still operating
without a strategic or financial plan to implement information technology
(Green, 1996)." Without knowing how to implement the technology into
the learning process, it is quite difficult to decide how to allocate or
reallocate what resources.
Oxford's Alex Reid points out that we haven't progressed much in the delivery
of good learning materials. "After 30 years of courseware development,
we still have very few examples that we can point to as models for future
development. ...Without ample examples of good practice in every discipline,
we will not progress (Reid, 1994)." As this study suggests, it is more
than the courseware materials, good models may intensely involve the delivery
system access, styles and methods as a significant part of the mix for developing
good examples to build upon.
Methodology
In order to assess students' perceptions of outcomes of various educational
media formats, an exploratory investigation was conducted in the context
of three different classes.
Participants
Traditional-age (e.g. 18-24) undergraduate students in three classes with
different combinations of educational media formats were surveyed.
Visual Communication (8 respondents): Half of class materials online; traditional
textbook; traditional lectures. Online materials were lecture notes and
Power Point presentations. Traditional
lectures were performed based on textbook material.
Computer Graphic Design (10 respondents): On-line supplementary materials,
traditional
textbook, no traditional lecture. Online material was supplementary and
included illustrated notes, Power Point presentations duplicated from class
lectures, and Web pages of linkages to relevant additional material. A traditional
textbook was used. Rather than traditional lecture, a coaching, hands-on
style of face-to-face teaching was used.
World Wide Web Publishing (16 respondents): All class materials online;
no textbook;
traditional lecture. Tutorials, lecture notes, Power Point presentations,
multimedia presentations, use of the WWW as a personal slide projector were
the online materials. Traditional lecture was performed in addition to material
presented online. There was no textbook.
Instrumentation
Questionnaires were designed to measure concept learning and reaction to
medium of presentation
in each of three classes. Questions were formatted with 7-point Likert scale
response options bounded by 1=very little and 7=very much.
Respondents in the the visual communication class assessed presentations
using on-line delivery
and, separately, assessed the use of PowerPoint in terms of concept learning,
concept clarity, concept comprehension, concept application, concept understanding,
and student comfort.
For the graphic design class, different combinations of educational media
were assessed: the
combination of text and on-line delivery; presentations based solely on
the text; and presentations using only on-line delivery. Questions were
constructed to assess the quality of educational experience of each format:
concept learning, concept clarity, concept comprehension, concept application,
concept understanding, and student comfort. Finally, students were asked
whether they could master the material without the text.
Respondents in the World Wide Web Publishing class, which used no textbook,
were asked to
assess the extent to which having a textbook would contribute concept learning,
concept clarity, concept comprehension, concept application, concept understanding,
and student comfort to their learning experience.
Additionally, students' overall impression of on-line delivery was assessed
for its enjoyment,
interest, helpfulness, stimulation of learning, and its ability to retain
attention. Ease of access to on-
line materials at home, school, and work was also assessed for this class.
Students were asked to
compare slide shows using Netscape to lecture alone; to lecture with overhead;
to lecture with Power Point, and to lecture with videotape. Finally, students
were asked to consider whether they could master the material without an
instructor.
Students in all three classes were asked whether on-line material puts too
much responsibility
for learning on the student.
Results
The goal of this study was to explore students' perceptions of educational
outcomes for various
educational media delivery formats. Both descriptions of attributes for
several formats and comparisons across formats were examined. Questions
specific to a class were analyzed separately. Total N may be reduced in
some cases due to missing data. Results are presented primarily in tabular
form to facilitate comprehension.
Graphic Design and Visual Communication: Description
and Comparison
Perceptions of on-line learning were assessed by combining the responses
from Graphic Design and Visual Communication to get an overall estimate.
Since learning goals (e.g. application vs. memorization) may differ in effectiveness
in the on-line format, perceptions of the on-line class format were compared
across these two types of classes: Visual communication is a theory-driven
class; Graphic Design focuses on application of principles and hands-on
learning.
Extent to which on-line material provides the following aspects (learning
variables) to
educational experience:
| Learning Variable (1) | Graphic Design (Mean) | Graphic Design (Std. Deviation) | Visual Comm. (Mean) |
Visual Comm. (Std. Deviation) | Combined (Mean) | Combined (Std. Deviation) |
| Learning of Concepts* | 4.50 | .85 | 5.50 | .76 | 4.94 | .94 |
| Clarity of concepts | 4.40 | .84 | 5.25 | 1.04 | 4.78 | 1.00 |
| Comprehension of concepts | 4.60 | .84 | 5.25 | 89 | 4.89 | .90 |
| Application of concepts * | 6.30 | .68 | 5.00 | .76 | 5.72 | .96 |
| Understanding of concepts | 5.12 | .97 | 5.13 | .84 | 4.83 | .92 |
| Student comfort with class | 5.00 | 1.05 | 5.50 | 1.20 | 5.22 | 1.11 |
| Learning variable (1) | Mean | Standard Deviation |
| Supplements learning material in textbook | 6.25 | 1.17 |
| Learning of concepts | 6.00 | .76 |
| Clarity of concepts | 5.63 | .74 |
| Comprehension of concepts | 5.50 | .54 |
| Application of concepts | 5.25 | 1.17 |
| Understanding of concepts | 6.25 | .71 |
| Student comfort with class | 5.38 | 1.19 |
| Learning variable (1) | Mean | Standard Deviation |
| Workload | 4.88 | 1.67 |
| Learning of concepts | 2.88 | 1.93 |
| Clarity of concepts | 2.94 | 1.48 |
| Comprehension of concepts | 2.88 | 1.50 |
| Application of concepts | 1.75 | 1.24 |
| Understanding of concepts | 2.56 | 1.37 |
| Student comfort with class | 2.69 | 1.62 |
Overall impression of on-line delivery of classroom material:
| Impression(1) | Mean | Standard Deviation |
| Enjoyment | 6.25 | .86 |
| Interesting | 6.19 | 1.17 |
| Helpfulness | 5.63 | 1.20 |
| Keeps attention | 5.88 | .96 |
Personal slide shows using Netscape were compared with other class formats
to assess the
utility of various combinations of classroom presentation methods.
Frequencies for question asking for a comparison of personal slide shows
using Netscape to
other instructional formats:
| Format | Netscape better | Netscape equal | Netscape worse |
| Lecture alone | 12 | 4 | 0 |
| Lecture with overhead | 13 | 3 | 0 |
| Lecture with Power Point | 5 | 10 | 0 |
| Lecture with videotape | 5 | 9 | 2 |
| Impression | Mastery without instructor | Too much responsibility on student |
| Enjoyment | .36 | -.51 |
| Interesting | .47 | -.71** |
| Helpfulness | .45 | -.70* |
| Keeps attention | .48 | -.61* |
| Responsibility | Mastery without instructor/text | Too much responsibility on student |
| Ease of access at home | .35 | -.29 |
| Ease of access at school | .21 | -.50 |
| Ease of access at work | .21 | -.20 |
Notes
Clark, R.E. (1983). "Reconsidering Research on Learning from Media."
Review of Educational
Research, volume 53, number 4, p. 445.
Clark, R. E. (1994). "Media Will Never Influence Learning." Educational
Technology, Research and
Development, volume 42, number 2, p. 21-29.
de Chardin, T., (1973). Towards the Future, Collins, p. 34-35.
Gillford, B. (1995). Educom Review, Nov./Dec., p. 36.
Farganis, S. and Dunn, R (1996). "Course Development," DIAL Conference,
New School, New
York, June 1996.
Green, K. (1966). "Annual Survey of Campus Computing," Claremont
(CA) Graduate School,
Claremont Consortium.. Available via cgreen@earthlink.net
Illich, L. (1970), Deschooling Society, Calder and Boyars, p. 125-156.
Kozma, R. B. (1994). "Will Media Influence Learning? Reframing the
Debate." Educational
Technology, Research and Development, volume 4, number 2, p. 14.
McLuhan, M., (1962). The Medium is the Message. Penguin. p. 34.
O'Donnell, James J. (1995). "Teaching with technology," PENN PRINTOUT,
University of
Pennsylvania Computing Center Magazine. Vol 11, No. 5.
Reid, A., (1994). "Perspectives on computers in education: the promise,
the pain, the prospect,"
Active Learning, (December), p. 11-12.
Sangster, Alan (1995). "World Wide Web - what it can do for education,"
Active Learning, (July), p.
7.
Wiley, P. (1996). "Using WWW to augment classroom instruction,"
Research Report, Brigham Young
University, Department of Computer Science. http://people/windley/windley.html.
Witherspoon, John (1995). "Assessment of online delivery systems at
19 Universities, " The Distance
Educator Vol 1. Number 3 - Fall 95.
Appendix A
Note: The general class/professor evaluation instrument used for each class
is both numeric and qualitative.
Students seldom fill in the open-ended portion of the "comment"
section, however, the information below is taken
from the two questions concerning the use of the new media and the use of
the textbook.
Appendix A
Teaching Effectiveness Questionnaire: Comment Sheet
Q 1: What aspects of this instructor's teaching were most effective?
"I like how Dr. Edwards brings his experiences and shares them. It
is especially good when he shows
these examples from the real world on the WWW."
"His use of materials was very beneficial. He provided us with his
notes online to better assist us in
class."
"Powerpoint presentations"
"It's nice to learn stuff and then have it explained and demonstrated
on the WWW and in the
PowerPoint. I really liked the real examples."
"PowerPoint with the handouts. Info. available on line."
"The use of the computer to help with the class lectures."
"The note sheets of his lectures, the handouts with the PowerPoints
and the examples on line."
Q - 3: Please comment on the textbook, activities, assignments and media
used in this course.
(Visual Communications 428/528)
"The textbook and the PowerPoint with the handouts were a great supplement
to the lecture."
"The book followed along well with the class, computer online and in
class computer presentations. It
was all very beneficial and the online notes were really great."
"Book was interesting and easy to understand. Using the computer in
class for presentations and the
online notes were best and made the book more real as the online examples
were from the real world."
(Computer Graphic Design 431/531)
"The textbook gathered dust because the online material and the Netscape
presentations were great.
The online helped review for exams especially. I hope there will be more
things like this in this and
other classes."
"Online media is excellent. Handouts are excellent. Netscape stuff
was excellent. Textbook sucks."
"Textbook was ok. Other stuff was better, real and meaning was clearer."
"Using the online examples makes the ideas clear. Textbook got me confused
and the examples are at
least two years old. New stuff and real examples is better."
WWW Publishing Class Survey
This survey assesses your opinions about different ways of presenting educational
material. For each
question below, please circle the number that corresponds to your opinion.
To what extent would having a textbook for this class add to the following
elements of the class?
Very little Very much
Workload ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Learning of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Clarity of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Comprehension of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Application of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Student comfort with the class ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Understanding of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
What is your overall impression of on-line delivery of classroom material?
Enjoy very much Do not enjoy
Enjoyment ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Interesting Very interesting Very uninteresting
----------------------------------------------------------------------------
1 2 3 4 5 6 7
Not very helpful Very helpful
Helpfulness ---------------------------------------------------------------------------
1 2 3 4 5 6 7
Stimulates learning Prevents learning
Stimulates learning ---------------------------------------------------------------------------
1 2 3 4 5 6 7
Does not keep attention Keeps attention
Keeps attention ---------------------------------------------------------------------------
1 2 3 4 5 6 7
How easily can you get access to the on-line materials via computer in the
following
ways?
Not at all Anytime
At home ---------------------------------------------------------------------------
1 2 3 4 5 6 7
Not at all Anytime
At school ---------------------------------------------------------------------------
1 2 3 4 5 6 7
Not at all Anytime
At work ---------------------------------------------------------------------------
1 2 3 4 5 6 7
Please compare the personal slide shows using Netscape to other class formats,
circling
the answer (1, 2, or 3) that corresponds to your opinion:
Slide shows using Netscape were (1) better; (2) equal to; (3) less helpful
than lecture alone.
Slide shows using Netscape were (1) better; (2) equal to; (3) less helpful
than lecture with
overhead.
Slide shows using Netscape were (1) better; (2) equal to; (3) less helpful
than lecture with
Power Point.
Slide shows using Netscape were (1) better; (2) equal to; (3) less helpful
than lecture with
videotape.
How confident are you that Not at all Absolutely
you could master this material ---------------------------------------------------------------------------
without an instructor being 1 2 3 4 5 6 7
present?
Do you think that Not at all Absolutely
on-line material puts too much ---------------------------------------------------------------------------
responsibility for learning
you?
Visual Communication Class Survey
This survey assesses your opinions about different ways of presenting educational
material.
For each question below, please circle the number that corresponds to your
opinion.
To what extent does the online material provide each of the following aspects
to your
educational experience?
Very little Very much
Supplements learning material ----------------------------------------------------------------------------
in the textbook 1 2 3 4 5 6 7
Very little Very much
Learning of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Clarity of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Comprehension of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Application of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Student comfort with the class ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Understanding of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Do you think that Not at all Absolutely
on-line material puts too much ---------------------------------------------------------------------------
responsibility for learning 1 2 3 4 5 6 7
you?
To what extent do the Power Point examples given in class provide each of
the following
aspects to your educational -------------------------------------------------------------------
experience?1234567
Very little Very much
Supplements learning material ----------------------------------------------------------------------------
in the textbook 1 2 3 4 5 6 7
Very little Very much
Learning of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Clarity of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Comprehension of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Application of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Student comfort with the class ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Increases understanding ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Graphic Design Class Survey
This survey assesses your opinions about different ways of presenting educational
material.
The following scales ask you to evaluate several combinations of classroom
delivery methods
with regard to several aspects of your educational experience. For each
question below, please
circle the number that corresponds to your opinion.
Consider presentations using both the textbook and online delivery. To what
extent does
this combination provide the following aspects to your educational experience?
Very little Very much
Learning of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Clarity of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Comprehension of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Application of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Student comfort with the class ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Understanding of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Consider presentations using only the textbook. To what extent does this
method provide
the following aspects to your educational experience?
Very little Very much
Learning of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Clarity of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Comprehension of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Application of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Student comfort with the class ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Understanding of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Consider presentations using only online delivery. To what extent does this
method
provide the following aspects to your educational experience?
Very little Very much
Learning of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Clarity of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Comprehension of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Application of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Student comfort with the class ----------------------------------------------------------------------------
1 2 3 4 5 6 7
Very little Very much
Understanding of concepts ----------------------------------------------------------------------------
1 2 3 4 5 6 7
How confident are you that Not at all Absolutely
you could master this material ---------------------------------------------------------------------------
without the text? 1 2 3 4 5 6 7
Do you think that Not at all Absolutely
on-line material puts too much ---------------------------------------------------------------------------
responsibility for learning 1 2 3 4 5 6 7
you?