Proceedings

Conference Overview

Call for Participation

Track Sessions

Advisory Committee

Featured Speakers

Conference Agenda

Workshops

Registration Form

Payment Information

Hotel Information

Travel Information

Exhibit Information

Previous Conferences

itconf@mtsu.edu
Seventh Annual
Mid-South Instructional Technology Conference
 Teaching, Learning, & Technology
The Connected Classroom
April 7-9, 2002

Using Blackboard to Survey Students at Midterm

By: Donna Austin, John Austin
Track 1 - Effective Integration of Technology into Teaching & Learning
Interest: Faculty :: Lecture/Presentation :: Level: All

Proceeding

ABSTRACT

Increased attention on the quality of teaching has meant an increase in the use of student surveys as a measure of perceived teaching quality. Many student surveys, which typically take the form of teacher and course evaluations, are conducted at the end of a school term and are more summative in nature.  Results are given to the instructor after the class is concluded making it impossible to incorporate feedback into the current course. We will share how we used the anonymous survey feature in Blackboard to survey our students at midterm in order to allow our teachers to collect formative feedback and to make crucial changes if indicated.

Introduction

The ultimate purpose of any student evaluation of teaching is the improvement of teaching, and implicit in that is the improvement of student learning (Ballantyne, 1997). While the literature on student evaluation of teaching is constantly growing, it usually focuses on the instruments used, with very little about the mode used to administer the survey. At Louisiana State University in Shreveport, a mid-term survey for students was introduced as an option to help instructors make necessary changes in their classroom practices during that same semester. While our faculty feel student surveys conducted at the end of the term are extremely important, many voiced this concern: What can I be doing "now" to improve my teaching and my student's learning?  Another problem with end-of-term surveys is that sometimes the most recent impressions are the most vivid for students so they might not be reflecting on the whole semester when completing such a survey. This is another reason to introduce a mid-term survey—to find out about a wider spectrum of the semester rather than only the end of the term.

Before we came up with this idea of using Blackboard's survey feature to conduct mid-term surveys, faculty members had no structured feedback mechanism with which to make changes in their teaching style. The results from our end-of-semester surveys (Student Instructional Report-II) are not returned until at least a month into the following semester. Additionally, these surveys are only administered in the fall for tenured faculty members unless the faculty members request that their students be surveyed in the spring as well.

Review and Background

The literature that has been published on student evaluations of teaching is vast and many works give detailed reviews of what has been covered (Cashin, 1990; Centra, 1993; Ramsden, 1992).  A review of the literature indicates many articles in this genre are generally concerned with how feedback is given to faculty (Brinko, 1993) how faculty deal with it (Busuttil, 1995) or how useful it is (Marsh & Roche, 1994).  Most studies consider particular aspects - reliability and validity of questionnaires, biases that might occur, etc. It is not the focus of this paper to examine these areas, other than to note that the consensus of opinion is that student ratings are generally valid, reliable and free of bias (Ballentyne, 1997). Evaluation is generally the last step during the semester process. According to Ballentyne (1997), few studies look at what happens after the evaluation as far as reporting the results to the students.  A prerequisite condition for teachers to make improvements to their teaching as a result of student feedback is that they consider student opinion worth listening too. Respect, care for students, and listening to what they have to say on teaching and other issues are considered fundamental aspects of good teaching (Brookfield, 1986; Centra, 1993; Greene, 1973; Taylor, 1995; Vella, 1994).

In fall 2001 we constructed a 28-question survey whose content we will discuss next.  The survey questions were typed into Blackboard using the Pool Manager so that it could be put into an exportable format, which is a zipped file, enabling it to be used by any faculty member who chose to participate.  We had two hands-on workshops in February 2002 so that faculty could actually import the survey into their Blackboard classes.  A detailed instruction sheet (See Appendix-I) was provided at the workshop as well as an instructor to help with each step.  We worked through the various steps of importing the zipped file into the various Blackboard classes.  The next step was to use the Assessment Manager to modify the survey and make it available for students.  During this step, faculty members could add, change or delete questions from the survey.  They could also use it just like it was originally designed.

Eighteen faculty members attended the two workshops in February.  This is about 13% of our full-time faculty. Only 8 instructors (6%) have actually administered the survey in their classes at this point.  Four more say they are going to administer it soon, and we presume six are not going to administer it. All of the instructors who have given the survey have said they will give or have given feedback to their students about the pedagogical decisions they took or will be taking as a result of feedback received from the survey. Ramsden & Dodds (1989) suggest communicating with students on changes that have been made so that students complete future surveys seriously. While providing this information to students may reinforce their views of the importance of the surveys, this study addresses the mode used to administer the surveys, the ease of importing the survey and whether or not the instructor receives helpful information. The main issue for the instructors is that this mid-term survey provides evidence of the instructor's concern for students by surveying them when changes can still be made in their classrooms.

The Survey Creation

We created the 28-question survey (See Appendix II) by combining various survey questions found on the web (e.g., Murdoch University Student Surveys of Teaching questions - http://cleo.murdoch.edu.au/evaluation/survey/teachdraft.html), along with some questions from Northwestern State University’s online course evaluation, and one question that pertains to a faculty member’s use of Blackboard.  There is one open-ended question at the end where students can make comments that relate to teaching and learning and the faculty member in general.  The student is warned to avoid comments that are personal, sexist or racist.  The student is also reminded that there will be another opportunity at the end of the semester to comment on the class again in an anonymous fashion.  Students do not take this survey using paper and pencil.

The students are reminded several times in the instructions and in the Survey Announcement Link that the survey is anonymous and that the instructor will not know who said what.  Results from surveys are reported as percentages.  Open-ended questions have simple listings of the typed comments.  Faculty members are reminded not to look at the survey results until an ending survey date has arrived. The instructor can see who took the survey, but not what the person said unless there is only one person who has answered the survey. On most questions, students answer using the following scale with these choices:  Strongly Disagree, Disagree, No Opinion, Agree, Strongly Agree and Unable to Judge.  Each faculty member can adapt the survey to his or her specific discipline, add or delete questions, or leave it in the more generic form.

Findings

After mid-term, we surveyed the 18 faculty members who attended the workshop to see if they used the survey.   We used a simple email survey, and all but two faculty members responded.  This survey is not meant to be used to show any statistical differences; it is simply reporting usage.  These are the questions we asked them.

  1. Did you actually import and use the survey after attending the class or looking at the survey in BBDEMO?
  2. Did you give it to your class once or more often?  If more often, how often? 
  3. Did you change the survey - add or delete questions?
  4. Did you find it easy to follow the directions you were given to IMPORT and TURN ON the survey?
  5. If #4 was NO, could you list some of the problems you had. If YES, skip this one.
  6. Did you give extra points to your students to motivate them to answer the survey?
  7. What was your response rate?  How many students actually took the SURVEY in your various classes?  (example: 25/39)
  8. Was the information you obtained from the survey helpful to you in improving teaching and learning in your classroom? If you would like to, please explain how you used some of the responses and what you did as a result of the responses from your students.
  9. Did you communicate to your students that you would use the information from the survey to improve teaching and learning?  The literature says this is important.

Eight of the faculty members had actually administered the survey at this time while four said they were going to do so soon.[Q1]  One had given it more than once.[Q2]  Some of those who had not yet given the survey or do not plan to give it did not answer some of these questions, so various answers will not always add to 16.  Six either did not change the questions or did not plan to change the questions as given, while five did make changes.[Q3]  All found the instructions easy to follow.[Q4]  Six gave extra credit (points) for participation, while four said they would not do so.[Q6]  Many did not report response rates, but those giving points had in the 70-85% response rates, while the others had response rates in the 30-60% range.  Nine said they had or would be communicating with their students about how the survey results would be used in the current class to improve teaching and learning.

Limitations, Implications and Conclusion

There are several implications that can be drawn from the present study. First, Black-board is fairly easy to use for anonymous surveys, but it is not robust or even very usable as far as data analysis is concerned.  Results cannot be sent to a statistical analysis program such as SPSS or SAS or even a database such as Microsoft Access. If you are satisfied with simple percentages for each answer, then Blackboard is sufficient.  Second, instructors have to be willing to wait until the time period has expired for the students to complete the survey before looking at the results.  Otherwise, they may be able to figure out what a “lone” person has said.  A sophisticated survey instrument would not have this problem. Third, students who accidentally get out of Blackboard while answering the survey cannot be “reset” to begin again like a quiz. You must have some “dummy” names for these students to use in this event.  Courses that are totally online courses, as opposed to those that use Blackboard as a component, should be distinguished as such within the survey and possibly have some unique questions.  According to Theall (2000), we know very little about the dynamics and influence of web-based and distance education vis-à-vis the evaluation of teaching in these contexts.  The roles, responsibilities, and tasks of teachers and learners in these online classes are different from those in face-to-face classes.  These differences should be explored and addressed in a future Student Survey of Teaching.  Finally, faculty who give mid-term evaluations and have consultation about them raise their scores to the 75th percentile on evaluations given at the end of the term (Cohen, 1980).  Faculty who only examine their mid-term survey results raise their scores to the 58th percentile from the 50th percentile of those giving no mid-term survey.  However, this paper did not address the use of faculty consultations.  Since it is quite obvious from a review of the literature that consultation is tremendously important in making improvements in student ratings in the end-of-term survey, the workshop to teach faculty “how to import” the survey into their Blackboard class should also include instruction and help about what to do with the survey results after the fact in order to get these kinds of improvements.

References

Ballantyne, C. (1997). Improving university teaching: Giving feedback to students. In Pospisil, R. and Willcoxson, L. (Eds), Learning Through Teaching, p12-15. Proceedings of the 6th Annual Teaching Learning Forum, Murdoch University, February 1997. Perth: Murdoch University.
http://cleo.murdoch.edu.au/asu/pubs/tlf/tlf97/ballan12.html

Brinko, K. T. (1993). The practice of giving feedback to improve teaching. Journal of Higher Education, 64(5), 575-594.

Brookfield, S. D. (1986). Understanding and facilitating adult learning. San Francisco, Jossey-Bass.

Busuttil, A. (1995). A study exploring the linkage between student evaluation of teaching and teaching development. Master of Higher Education Thesis. University of New South Wales.

Cashin, W. E. (1990). Assessing teaching effectiveness. How administrators can improve teaching.  P. A. Seldin. San Francisco, Jossey-Bass.

Centra, J. A. (1993). Reflective faculty evaluation. San Francisco, Jossey-Bass.

Cohen. P. A. (1980). Effectiveness of student ratings feedback for improving college instruction: A meta-analysis of findings. Research in Higher Education, 13(4), 321-341.

Greene, M. (1973). Teacher as stranger. Belmont, California, Wadsworth Publishing Co.

Marsh, H. W. and L. A. Roche (1994). The use of students' evaluations of university teaching to improve teaching effectiveness. Canberra, DEET, Higher Education Division, EIP, AGPS.

Ramsden, P. (1992). Learning to teach in higher education. London, Routledge.

Ramsden, P. and A. Dodds (1989). Improving teaching and courses: A guide to evaluation. Melbourne, Centre for the Study of Higher Education, University of Melbourne.

Richlin, L. R. and M. D. Cox (1991). The scholarship of pedagogy: a message from the editors. Journal on Excellence in College Teaching, 2: 1-8.

Taylor, P. G. (1995). Initial teacher education: the pedagogy is the message. South Pacific Journal of Teacher Education, 2(2), 207-216.

Theall, M. (2000).  Electronic course evaluation is not necessarily the solution.  The Technology Source.  Retrieved March 29, 2002, from the World Wide Web: http://ts.mivu.org/default.asp?show=article&id=823

Vella, J. (1994). Learning to listen, learning to teach. San Francisco, Jossey-Bass

Appendix - I

Directions for getting the survey and installing it in your Blackboard class

To SAVE to your hard drive the documents you will need:

  1. Go to the BBDEMO class.
  2. Click on Course Information.  Here you will find the zipped file that you will need to make the survey for YOUR Blackboard classes. This is file #1 and is called Export_StudentSurveyofTeaching.zip. There is also a document that has the words prepared for you to use if you want in actually making your survey available to your class.  It is file #2 under Course Information.
  3. Double-click on the zipped file and SAVE to a folder on your hard drive that you will remember.  Do the same for the document called “Information to use when Preparing Student Survey of Teaching.”

Now you have two files saved on your hard drive that you will use in the following steps.

To IMPORT this zipped file into your Pool Manager:

  1. Go to YOUR Blackboard class where you want to give the survey
  2. Click on CONTROL PANEL
  3. Click on POOL MANAGER
  4. Click on IMPORT POOL
  5. Click on the BROWSE button and find the folder where you saved the zipped file.
  6. Double-Click on this file.  Then press SUBMIT.
  7. You should see this file on your screen in a list.  It’s now been imported.
  8. Click on RETURN TO CONTROL PANEL.

To make the survey for your class using Assessment Manager:

  1. You should be in Control Panel.  Click on ASSESSMENT MANAGER.
  2. Click on Create New Assessment.
  3. A box appears that says “Type of Assessment.”  There is a dropdown box.  Click on the dropdown arrow and select SURVEY.  Click SUBMIT.
  4. Type Student Survey of Teaching in the first box that says “Enter a Name:” or type whatever you want to call your survey.
  5. Type a description in the box that says “Enter a Description.”  The next paragraph is what I type into this box. [You can edit this paragraph or type your own.] The info in this box will appear with the link in the announcement telling the student about the survey.  [See BBDEMO to see how the announcement link looks.] 

Please take this survey for participation points.  It is anonymous!  I will know that you took it, but not what you said. 

  1. Click Submit.
  2. The next box says “Enter Instructions.”  The next paragraph is what I use here.  Use this or type your own.

This is an anonymous survey.  Only I will see the results but I will NOT know who said WHAT, only that you responded to the survey.  I cannot match up your name with your answers. Please respond to the questions and press the Submit Answers button at the end when you have completed all 28 questions.

  1. Click SUBMIT.
  2. The next screen says, “Select a Question Type.”  Click on the dropdown arrow and SELECT “From Question Pool or Assessment” at the bottom and then click SUBMIT.

10.     Put a check mark to the left of Student Survey of Teaching – 28 questions and then click the SUBMIT button at the lower right.  You don’t need to preview at this time.

11.     The next screen will have “ALL” selected for “Related Categories” and “Select Question Type.”   Leave those selections as they are and click SUBMIT.

12.     The next screen says “Search Results.”  You will need to check (click each box) all of the 28 questions.  If you do not want to use one of the questions in your survey, then do NOT check off that question.  Do NOT delete Question #5 since that is where I’ve explained what the selections mean, e.g., SA = Strongly Agree.  You will have to scroll down the page to check all 28 questions.  At the bottom of this page, click SUBMIT to select all questions for YOUR survey.

13.     At this point, I make the first essay question (it’s really an open-ended question) number 28 by clicking on the drop-down arrow with #1 and selecting #28 instead.  I want the open-ended question to be the last question.  I also make the #2 question about Blackboard usage to be #27.  That’s just my preference. 

14.     You can add your own questions at this point by click on the “Add New Question” button at the upper-left side of the screen.

15.     You are now ready to click on the SAVE AND MAKE AVAILABLE button.

16.     Click on the YES button in the block that says “Make Assessment Available.”  Leave Generate an Announcement? as it is with YES checked.  Leave “Place a Link in Assignments” as it is.

17.     DO NOT allow multiple attempts.  Each student should answer only once.

18.     Scroll down and click the SUBMIT button.

19.     Your survey has now been created.  Click on the Return to Course button.

To View the Results of the Survey:

  1. Click on Control Panel
  2. Click on Online Gradebook
  3. Click on Report by Item
  4. Click the dropdown arrow and Select Student Survey of Teaching.  Click GO.
  5. Click on VIEW ITEM ANALYSIS
  6. To see the answers to the last open-ended question, go to that question and click on VIEW RESPONSES.
  7. Click the BACK button at the bottom left to return to where you were.
  8. Click on RETURN TO COURSE.

Appendix - II

Student Survey of Teaching

Instructions:  This is an anonymous survey.  Only your teacher will see the results but he or she will NOT know who said WHAT, only that you responded to the survey.  He or she cannot match up your name with your answers.  Please respond to the questions and press the Submit Answers button at the end when you have completed all 28 questions.

  1. The following were used as resources in this class:  (check all that apply)
    1. the textbook
    2. PowerPoint
    3. A computer
    4. A projector hooked to a computer
    5. An overhead projector
    6. The Internet
    7. Other reading materials (articles, etc.)
    8. Other
  2. This class has added to my skills in the following ways: (check all that apply)
    1. Solving problems
    2. Writing papers
    3. Designing lab experiments
    4. Finding trends in data
    5. Critically reviewing articles
    6. Working effectively with others
    7. Giving oral presentations
    8. Learning more about computer programs
    9. Other
  3. In this teacher’s classes, I have gained a good understanding of the concepts covered.  (The choices are for questions 3 through 26 but will be listed here only once.)
    1. Strongly disagree
    2. Disagree
    3. No Opinion
    4. Agree
    5. Strongly Agree
    6. Unable to Judge
  4. This teacher explains the purpose of each class as it relates to the unit.
  5. This teacher is well prepared for class.
  6. Classes by this teacher are well organized.
  7. This teacher communicates effectively with students.
  8. This teacher demonstrates enthusiasm for the subject.
  9. This teacher encourages student participation in class and/or groups.
  10. This teacher provides me with opportunities to apply what I learn.
  11. This teacher encourages me to be responsible for my own learning.
  12. This teacher is sympathetic to student differences.
  13. I found this teacher helpful if I encountered difficulties in this unit.
  14. It is clear what I am expected to learn in each class.
  15. This teacher provides me with useful feedback.
  16. Work graded by this teacher is returned in a reasonable time frame.
  17. This teacher marks assigned work fairly.
  18. The course syllabus was clear and unambiguous.
  19. Method(s) of evaluation to be used were clearly explained.
  20. The subject matter covered thus far has been well organized.
  21. Difficult topics have been clearly explained.Student questions have been encouraged.
  22. Resources required for the course were available.
  23. Important topics of the course have been included on tests and/or projects.
  24. Feedback about student performance on course assignments has been provided.
  25. Instructional quality in this course has met my expectations thus far.
  26. Referring to our course management tool, Blackboard, (check all that apply)
    • My teacher does NOT use Blackboard
    • It’s used very little
    • It’s used a great deal by this teacher
    • Blackboard helps me to stay on track
    • Blackboard is a good thing for us to have
    • I don’t like Blackboard
  27. Comments relating to the teacher can be typed here.  There will be an opportunity to comment on the class overall at the end of the semester with another instrument.  This is an anonymous survey.  The teacher will see the results but won’t know who said what.  Please confine your comments to issues relating to teaching and learning.  Any comments that are personal, sexist or racist will be destroyed.  Additionally, what are one or two things that this teacher does that help you learn?