Abstract
Introduction
Purpose of the Project
Survey Results
Results from Student Evaluation of
Instruction (IDEA)
Results from Actual Student
Performance
Conclusions
The use of the World Wide Web for the delivery of course information is growing at a rapid rate. However, little research has been done to assess the impact of Web-based instruction upon student attitude and performance. This paper describes a project undertaken to assess the actual and perceived impact of the Web upon student learning. Results indicate that the use of the Web improves students' attitude toward the course. However, no evidence was found to support the claim that Web-based instruction actually improves actual or perceived student performance.
The use of the World Wide Web for the delivery of course information is growing at a rapid rate. The degree of utilization ranges from simple on-line course syllabi to detailed on-line courses designed for distance learning. Some research has been done to determine student attitude toward the Web, such as Lehman and Lehman ( 1996) and Takle and Taber ( 1996). Less has been done in the area of assessing impact of Web-based instruction upon student learning. Universities are clearly interested in determining the impact of technology upon student learning. A story in the February 19, 1998 on-line issue of the Chronicle of Higher Education describes a survey developed by a non-profit organization, Teaching Learning Technology Group (affiliated with AAHE), that enables universities to determine technology's impact upon student learning
The purpose of this paper is to present the initial results of a project undertaken in the Department of Economics at Southeast Missouri State University designed to determine the impact of Web-based instruction upon student attitude and performance in freshman and sophomore level economics courses. In its Strategic Plan developed in September 1995, the department cited "increased usage of Internet resources" for classroom instruction as one of its objectives. The utilization of technology as an integral component of instruction is also been stated as a priority of the Harrison College of Business, in which the Economics department is housed, and of the University. Competency in the use of the Internet is a necessary skill that all students should develop before graduation; ideally, this competence will be developed early, enabling them to use the Internet for information gathering throughout their college career. Many university students grew up with the Internet, and may respond positively to efforts to include this technology in the classroom. The technological facilities in the recently constructed Robert Dempster Hall allow faculty to incorporate the Internet directly into classroom activities. Labs throughout campus allow students to use the Internet to complete out-of-class assignments.
The purpose of this project was to increase student interest in 100 and 200 level economics courses, and reduce the percentage of Ds and Fs earned in these courses, by incorporating the Internet (more specifically, the World Wide Web) into economics courses. During the past five years, the percentage of low grades earned in EC 101 (Economic Problems and Policies), EC2 15 (Principles of Microeconomics), and EC225 (Principles of Macroeconomics) ranged from seventeen percent to almost thirty percent. Results from student evaluations and conversations with students cite low student interest as a problem. Increasing student interest may also lead to an increase in retention, which is a goal of the University.
According to the plan of the project, participating faculty would have websites completed by Fall 1997 and would use the Web in courses during that semester. Web-based instructional material included on-line course syllabi, on-line tutorials, links to course related resources on the Web, and bulletin boards. Three instructors, teaching seven sections of lower level economics courses, were identified as "users, incorporating the Web at least eight times in class and requiring students to use it for at
least one assignment. Four instructors, teaching eleven sections of lower level economics courses, were identified as "non-users," incorporating the Web less than three times during class and not requiring its use for out of class assignments. It is worth noting, however, that 49% of students in the "non-user" sections reported that they did utilize the World Wide Web at least one time in preparing assignments.
The most extensive source of information regarding student attitudes toward Web usage in economics classes was obtained from two surveys that were distributed to all sections of EC 101, EC215 and EC225 during the Fall 1997 semester. A pre-survey was distributed during the first week of class, and a post-survey during the last week of class. 269 students in the "user" classes completed the pre-survey, and 169 completed the post survey. 307 students in the "non-user" classes completed the pre-survey, and 239 complete the post-survey.
Surveys were processed according to whether or not the instructor was a "user" of the Web. As tables 1 through 3 show, results of the pre-survey show that there was no statistically significant difference between students in the "user" classes and "non-user" classes with respect to their initial opinions about the value of the World Wide Web for instructional purposes.
On the pre-survey, students were asked to respond to the following statement: "Using the World Wide Web in class or for class assignments has enhanced my learning." As the information in the following table shows, there was little difference in responses between "user" classes and "non-user" classes.
Table l
| Users | Non-users | |
| Strongly Agree | 21% | 15% |
| Agree | 44% | 46% |
| Not sure | 28% | 27% |
| Disagree | 5% | 7% |
| Strongly Disagree | 2% | 4% |
In the user group, 65% of students agree or strongly agree with this statement, while in the non-user group, 61% of students agree or strongly agree with this statement. Testing the null hypothesis of no difference between percentages (Ho: Pu = Pn HA:Pu = Pn), a z value of .99 leads one to fail to reject the null hypothesis of no difference. Well over half of the students believed that the Web had enhanced their learning in other classes.
On the pre-survey, students were asked to respond to the following statement: "Using the World Wide Web in class or for class assignments makes the class more interesting. "
Table 2
| Users | Non-users | |
| Strongly Agree | 21% | 15% |
| Agree | 44% | 45% |
| Not sure | 22% | 26% |
| Disagree | 11% | 11% |
| Strongly Disagree | 2% | 3% |
In the user group, 65% of students agree or strongly agree with this statement,
while in the non-user group, 60% of students agree or strongly agree with
this statement. Testing the null hypothesis of no difference between percentages
(Ho: Pu =
Pn HA:Pu =
Pn),
a z-value of 1.32 leads one to fail to reject the
null hypothesis of no difference. Well over half of the students believed
that the Web had made classes more interesting.
On the pre-survey, students were asked to respond to the following statement: "Teachers should required World Wide Web assignments for their classes."
Table 3
| Users | Non-users | |
| Strongly Agree | 9% | 8% |
| Agree | 26% | 25% |
| Not sure | 29% | 33% |
| Disagree | 29% | 25% |
| Strongly Disagree | 7% | 10% |
In the user group, 34% of students agree or strongly agree with this statement, while in the on-user group, 33% of students agree or strongly agree with this statement. Testing the null hypothesis of no difference between percentages (Ho: Pu = Pn HA:Pu = Pn), a z-value of .66 leads one to fail to reject the null hypothesis of no difference.
Results of the post-survey, found in Tables 4 and 5, show that there was a statistically significant difference between students in the "user" classes and "non-user" classes with respect to their opinions about the value of the World Wide Web in their economics classes. Over three-quarters of the students in the "user" classes found the WWW to be a valuable instructional tool.
On the post-survey, students were asked to respond to the following statement: "Using the World Wide Web in my economics class has enhanced my learning."
Table 4
| Users | Non-users | |
| Strongly Agree | 17% | 5% |
| Agree | 63% | 27% |
| Disagree | 12% | 8% |
| Strongly Disagree | 5% | 2% |
| Not applicable | 2% | 58% |
In the user group, 80% of students agree or strongly agree with this statement, while in the non-user group, 32% of students agree or strongly agree with this statement. Testing the null hypothesis of no difference between percentages (Ho: Pu = Pn HA:Pu = Pn), a z-value of 4.77 leads one to reject the null hypothesis of no difference at the .001 level. There is a very high level of support for use of the WWW by students in the "user" classes, and, not surprisingly, this support is significantly higher than that in the "nonuser" classes. It is also important to note that well over half of the students in the "nonuser" classes who used the WWW on their own found that it enhanced their learning. (According to post-survey survey results, 49% of students in the "non-user" classes used the WWW at least once for assignments in their economics classes.)
On the post-survey, students were asked to respond to the following statement: "Using the World Wide Web in my economics class has made my economics class more interesting. "
Table 5
| Users | Non-users | |
| Strongly Agree | 20% | 8% |
| Agree | 63% | 22% |
| Disagree | 12% | 12% |
| Strongly Disagree | 4% | 2% |
| Not applicable | 1% | 55% |
In the user group, 83% of students agree or strongly agree with this statement, while in the non-user group, 30% of students agree or strongly agree with this statement. Testing the null hypothesis of no difference between percentages (Ho: Pu = Pn HA:Pu = Pn), a z-value of 5.27 leads one to reject the null hypothesis of no difference at the .001 level. Again, there is a very high level of support for use of the WWW by students in the "user" classes, and, not surprisingly, this support is significantly higher than that in the "non-user" classes. It is also important to note that well over half of the students in the "non-user" classes who used the WWW on their own found that it made the class more interesting.
Results of the pre-survey (Table 6) show that there was no statistically significant difference between students in the "user" classes and "non-user" classes with respect to their initial opinion about whether or not they expect their economics class to be boring. However, there was a difference after students took their economics class (Table 7).
On the pre-survey, students were asked to respond to the following statement: "I expect my economics class to be boring and irrelevant."
Table 6
| Users | Non-users | |
| Strongly Agree | 3% | 2% |
| Agree | 5% | 8% |
| Not sure | 41% | 40% |
| Disagree | 43% | 40% |
| Strongly Disagree | 9% | 9% |
In the user group, 8% of students agree or strongly agree with this statement, while in the non-user group, 10% of students agree or strongly agree with this statement. Testing the null hypothesis of no difference between percentages (Ho: Pu = Pn HA:Pu = Pn), a z-value of .69 leads one to fail to reject the null hypothesis of no difference.
On the post-survey, students were asked to respond to the following statement: "My Economics class has been boring and irrelevant."
Table 7
| Users | Non-users | |
| Strongly Agree | 3% | 10% |
| Agree | 8% | 15% |
| Not sure | 19% | 15% |
| Disagree | 56% | 47% |
| Strongly Disagree | 14% | 14% |
In the user group, 11% of students agree or strongly agree with this statement, while in the non-user group, 25% of students agree or strongly agree with this statement. Testing the null hypothesis of no difference between percentages (Ho: Pu = Pn HA:Pu = Pn), a z-value of-3.62 leads one to reject the null hypothesis of no difference at the .001 level. A significantly larger percentage of students in the "non-user" classes found their economics class to be boring and irrelevant. Of course, it is not possible to attribute this difference solely to the use of the World Wide Web. There may be other differences in teaching style between the "user" and "non-user" instructors, and differences in learning styles between students in "user" and "non-user" classes, that account for this difference. Yet, it is an interesting result that bears further investigation.
Did lack of access to computer facilities affect the responses of the two groups? Evidence from the pre- and post-surveys (Table 8) indicate that it did not.
On both surveys, students were asked to respond to the following statement: "I can conveniently obtain access to the World Wide Web using my own computer or a computer in the student labs or dorms on campus."
Table 8
| User Pre-survey | Non-users Pre-survey | Users Post-survey | Non-users Post-survey | |
| Strongly Agree | 31% | 29% | 45% | 49% |
| Agree | 42% | 45% | 41% | 35% |
| Not sure | 11% | 12% | 3% | 5% |
| Disagree | 10% | 11% | 9% | 8% |
| Strongly Disagree | 6% | 4% | 2% | 4% |
On the pre-survey, 73% of students in the "user" classes agreed or strongly agreed with this statement; the corresponding percentage in the "non-user" classes was 74%. On the post-survey, 86% of students in the "user" classes agreed or strongly agreed with this statement, while the percentage in the "non-user" classes was 84%. Clearly, there is no difference between the two groups on either survey.
Did use of the World Wide Web have an impact on students' perception of their grades? On the pre-survey and post-survey, students were asked to respond to the following statement: "I expect my grade in economics this semester to be a(an):" As Table 9 shows, on the pre-survey, there was no real difference in student's expectations of their grades. 93% of students in the "user" classes expected to receive an A or a B, while 94% of students in the "non-user" classes expected to receive an A or a B. Likewise on the post-survey, 61% of students in the "user" classes expected an A or a B, while 63% of students in the "non-user" classes expected an A or a B.
Table 9
| Users Pre-survey | Non-users Pre-survey | Users Pre-survey | Non-users Pre-survey | |
| A | 47% | 53% | 25% | 10% |
| B | 46% | 41% | 36% | 44% |
| C | 6% | 6% | 34% | 33% |
| D | 1% | 0% | 5% | 4% |
| F | 0% | 0% | 0% | 1% |
On the pre-survey, there was no real difference in student's expectations of their grades. 93% of students in the "user" classes expected to receive an A or a B, while 94% of students in the "non-user" classes expected to receive an A or a B. Likewise on the post-survey, 61% of students in the "user" classes expected an A or a B, while 63% of students in the "non-user" classes expected an A or a B.
Results from Student Evaluation of Instruction (IDEA)
All instructors in the Department of Economics used the IDEA student evaluation form in the Fall 1997 semester. There is a percentile score reported for all instructors regarding students' perception of their "improved attitude toward field." The weighted (according to number of students in class) average percentile for instructors using the WWW was 54, while the weighted average percentile for those instructors not utilizing the WWW was 30. Again, it is not possible to attribute this difference solely to use of the WWW, but it is a result that is worth further investigation.
Results From Actual Student Performance
It is not possible to compare the grade distribution of "user" classes and "nonuser" classes for a variety of reasons, the most obvious being that most of the "non-user" classes were EC215, while the majority of "user" classes were EC 101. Grades tend to be somewhat lower in EC2 15 than in EC 101.
We might look at the grade distribution of individual instructors to determine if use of the WWW resulted in fewer Ds and Fs. Because of confidentiality issues, individual grade distributions are not reported here. However, an examination of the data indicate no significant change in the number of Ds and Fs after instructors incorporated the WWW into their classes.
Tentative results of this study indicate the use of the World Wide Web in economics classes makes economics more interesting to students and, according to students, increases their learning of economics. Results from student evaluations of instruction tend to support the hypothesis that utilization of the WWW improves student attitude toward economics. However, students' expectations of grades and actual grades given to students do not support the hypothesis that this increased interest results in improved performance.
The initial results of this study are interesting enough to warrant further investigation. The pre- and post- surveys are being distributed to all lower level economics classes in the Spring 1998 semester in order to increase the size of the sample. Further analysis will be conducted to separate the impact of WWW usage upon student attitude and performance from the impact of other instructional techniques. Also, differences in attitudes toward the WWW by gender, age, and year in school will be examined.
Lehman, Lisa M. and John A. Lehman, "Student Reactions to a World Wide Web-Based Textbook: An Exploratory Study," WebNet96 Proceedings Online, Association for the Advancement of Computing in Education.
Takle, Eugene and Michael R. Taber, "Use of the Web as a Tool for Interactive Learning," WebNet96 Proceedings Online, Association for the Advancement of Computing in Education.