Peer Review is the final step in the process of ensuring the success of course development and delivery. Once a determination of course standards is completed, employing a Peer Review process can check for quality and accuracy as well as how the student experiences the course and how the course design anticipates the achievement of learning outcomes for the course. This paper reviews the process used by Volunteer State Community College, from the development of course standards for Distance Learning Courses to the development of a formal Peer Review process of Distance Learning Courses. This review discusses what has been learned, the methods employed and what is working (or not working) for our institution.

Student Evaluations

The first step in course review is the student evaluation. At Volunteer State Community College, faculty was asked to develop a set of standards for distance learning, while utilizing the current student evaluation for courses. Although there should not be a substantive difference between the course content presented in the face-to-face course room and the distance course room, challenges arise (in the evaluation process), due to the nature of the delivery format.

 

A result included a revised student evaluation form to better address the differences in delivery for distance learning courses and the development of standards are to be applied to all distance learning courses.

 

In order to review this evaluation process, students were asked to participate in a focus group to provide their perspectives on the student evaluation form questions, distance faculty members were interviewed and a survey of existing literature was conducted.

Course Standards

The second step was to develop Standards. According to Miriam-Webster’s Dictionary online a definition for standards includes: something set up and established by authority as a rule for the measure of quantity, weight, extent, value, or quality. The Distance Learning Committee at Volunteer State Community College was given the task of developing a set of comprehensive standards for all Distance Learning Courses.  These standards were based on the student evaluation of all courses. Five (5) key areas included: 

Comprehensive standards were developed which would assist faculty members in determining how well their course was meeting the pedagogical and technological needs of the students. Each standard includes Musts – those items that every course absolutely had to include and Shoulds – those items that it would be nice for a course to include.

After the distance course standards were completed, reviewed and endorsed by the Distance Learning Committee, the Deans for each division and the Vice President of Academic Affairs provided their support and assistance for incorporating this improved method of looking at instruction. (for specifics, see the 2002 Conference Proceedings paper: “From Key Handouts to More Hands On Keys:  Planning for the Progressive Use of Technology by Faculty,” Nunaley and Warner.)

Course Peer Reviews

The final step in the process was to create a Peer Review process that would provide faculty members feedback throughout the creation of a new distance learning course. Each faculty member is to be assigned a “Peer Review Team.” The team includes: (1) a member with a strong technology background (to look at design, course structure and accessibility issues) and two content reviewers, (2) (one of which may be a professional from the community). Content reviewers can be from industry, in the case of AAS programs, or members from the college or other professional association with, individuals who possess a special expertise in the content area. (3) The second content reviewer (and third member of the team) will be a faculty member at the college.

 

The Peer Review process was designed to provide assistance before the course was officially offered to students. The conduct of Distance Course Peer Reviews will proceed, with the five (5) Standards areas providing a review framework. (See appendix for a sample Peer Review Form).

 

At Volunteer State CC, a Peer Review process exists for evaluating face-to-face teaching. This process provides feedback to the faculty member concerning course delivery. The Peer Review Process for distance courses will provide feedback during course development. In creating this change to the system, several areas of concern presented themselves:

How will faculty react to their course being reviewed before the course has been taught?

Who will appraise the Peer Review process?
What guidelines will be provided to Peer Reviewers?
Who will be the Peer Reviewers?
When will the Peer Reviews occur?
Will Peer Reviewers be rewarded (compensated) for their participation in the process? [s1]  

What are some obstacles to this process?

What materials, support and supplies will be required to complete this process?

The Distance Learning Committee is a college-wide standing committee with representatives from different divisions and departments. The Director of Distance Learning is an ad hoc member. The committee examined the issues noted above, and subsequently, spent numerous hours researching, discussing and debating the individual issues before adopting a format that would serve our institution and faculty.


How will faculty react to their course being reviewed before the course has been taught?

One of the key topics was how faculty would react to having their course peer-evaluated because a new face-to-face course does not require a prior and formal evaluation before a course begins. The Distance Learning Committee determined that it was critical to review a distance course before offering the course because of the many challenges faced by both the instructor and the student. The faculty member who is developing the course will know up front that the Peer Review process is a partnership designed to provide the best possible outcomes for all involved.


Who will appraise the Peer Review process?

Other distance learning instructors, department Chairs, division Deans, and ultimately the Vice-President for Academic Affairs will appraise the Peer Review process. The success (or failure) of a Peer Review process can also be validated by student evaluations and whether a student has achieved the desired learning outcomes.

What guidelines will be provided to Peer Reviewers?

The issue of guidelines for Peer Reviewers was also addressed. A short handout (see Appendix A) [s2]  was created to help focus the reviewer. The five key areas from the Course Standards were chosen for Peer Review of a distance course. Each aspect of the distance course will be subject to an examination for compatibility with the standards for Course Syllabus, Course Calendar, Communication, Materials/content and community building.

Feedback is to be provided the faculty course developer throughout the design phase so the course developer has the opportunity to incorporate any design or technical changes before offering the course to students. Each Peer Review form also provides space for additional comments and suggestions.


Who will be the Peer Reviewers?

Another key part of the process is to designate a reviewer who looks specifically at technical issues. ADA compliance, the proper use of technology and links that are accurate and working are examples of review areas. Other reviewers are looking for accuracy of content, ease of use, flow of the course (structure) and how the course is designed to encourage student interaction and community building.

Distance Learning Committee members also discussed the possibility of a recruiting several student reviewers. The student reviewer would review the course from the student’s perspective and look only at course layout and ease-of-use. The choice of members of the review team is accomplished by consulting with the course developer and by seeking additional recommendations from the appropriate division Dean.


When will the Peer Reviews occur?

Ideally, the Peer Reviews will occur during the development stage of the course. By providing feedback throughout the development stages, potential problems in areas such as accessibility, communication and ease of use is correctable before students encounter difficulties.

Existing courses should receive a Peer Review. Faculty currently teaching a distance course can benefit from a Peer Review. For courses that have used the distance format for multiple semesters, a reasonable time for a course Peer Review could be with the adoption of a new text or new edition to a text.

For new courses, this process should occur within one year of the initial course offering. This will allow the developer and the Peer Review team and opportunity to improve the course based on both student evaluations and Peer Review comments.

How will Peer Reviewers be rewarded (compensated) for their participation in the process?

The issue of compensation for Peer Reviewers is a “hot” topic. In an environment where budgets are growing ever tighter, additional financial compensation is not always practical.

Possible forms of compensation include additional release time, viewing Peer Review as the equivalent of committee service, or counting service as a Peer Reviewer as service “above and beyond the call of duty” for the college when promotion and/or tenure packages are being prepared. Many faculty members are troubled with the prospect of becoming a course Peer Reviewer because of the extra time requirements.

Some form of acceptable compensation will need to be in place in order to make the Peer Review program, not only viable, but also worthwhile for the faculty time commitments. The use of Peer Reviewers from outside the academic institution (content professionals) will also need institutional discussion. What form of recognition and/or compensation will be required for these members of review teams?


What are some obstacles to this process?

There are several obstacles to instituting a course Peer Review process. One of the greatest obstacles is the lack of support from faculty. Faculty may be apprehensive they are receiving a critique and that the Peer Review process may interfere with their academic freedom. Other faculty may have reservations related to the time commitment involved if they become a Peer Reviewer. Training is a second area of concern that requires the attention of the institution. How will Peer Reviewers receive training? Will faculty trainers come from the faculty pool, who are already teaching at a distance or will trainers be faculty members who are content experts but have little or no technology training? Should the Peer Reviewer who is not currently teaching a distance course still have had the experience of being a student in a distance course (to effectively evaluate a distance format course)? A third obstacle is funding. Will there be money available to provide training? Should or will additional compensation be provided for Peer Reviewers? How will content experts from outside the college be approached and should or how will they be compensated for their time? In many instances, the obstacles may seem so overwhelming that the entire project is dropped. Since Peer Reviews strengthening a course is a worthy goal, a thorough analysis of obstacles should be accomplished, and each institution should encourage creative solutions.

What materials, support and supplies will be required to complete this process?

In order to accomplish the Peer Reviews of distance courses, whether existing courses or courses proposed for development, an institution will need to plan to provide Peer Reviewers specific support.

The Director of Distance Learning must provide guidance to Peer Review Teams. This guidance will entail discussions of an acceptable time frame for completion of the process, areas of responsibility for each course reviewer, provisions for on-going feedback to the faculty course developer or current instructor and a discussion of the objectives and purpose of the Peer Review process.

WebCT is the institution-chosen course management program at Volunteer State CC. Peer Reviewers must be familiar with basic navigation of a WebCT-based course. This knowledge will be especially important if a Peer Review member is chosen from outside the institution (as a content specialist) or is a faculty member is chosen for the course review if the faculty member does not currently teach a distance course (using the WebCT course management program).  Other institutions may use a different course management program, such as BlackBoard, and may need to plan for navigation training for course reviewers.

Providing specific guidance to course Peer Review Teams should also include the specifics of course standards adopted by the institution. Course standards may be provided at an online location or by other means, such as disk, CD or paper copy.  In addition to a review of course standards as guidelines, Peer Review team members should be provided an evaluation instrument.  The instrument may be in paper or electronic form.

Finally, consideration should be given to the issue of type of access to a faculty member’s course.  Since Peer Review team members must be able to view the course materials and to tour course site, access (including passwords) must be arranged with the WebCT (or other course management program) Administrator. Arrangements for course access should be agreed upon in advance. 

Conclusion

The Peer Review process is a worthy [s3]  if not essential undertaking. Consideration must be given to the resources required (training, materials, compensation, and time) and the atmosphere and manner in which it is presented to faculty. The Peer Review process when done properly, can increase student retention in distance classes, increase course developer satisfaction by providing a system for continuous feedback and review and provide the institution with a method for assessing additional training needs for distance courses. Any institution considering adopting a Peer Review process must spend time looking at existing standards and evaluation formats and should conduct a needs assessment to determine the assistance faculty members might require. In order to be a success, the Peer Review process must have the full support of the Chief Academic Officer, Division Deans and faculty currently teaching at a distance.

 

Appendix A: Distance Learning Peer Review Analysis Form

 

Section 1 – Syllabus

Detailed Course Syllabus             Available on WEB site   Includes Course Description

Objectives = on-ground section   Clear Course Requirements

Contains Text Requirements                         Includes Technology requirements

Comments:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           

Section 2 – Schedule/Calendar

Structure Encourages timely assignment completion Establishes clear deadlines

Easily understood and followed directions                Course Materials match explanations

Includes agenda for any scheduled meetings           

Equitable distribution of assignment within semester

Comments:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           

Section 3 – Communication

Course provides “Teaching at a Distance” communication devices to engage students

Communication methods included in syllabus

Provides adequate student/faculty and student/student access

Timeliness on evaluations noted Office Hours and Office Phone noted

Email address noted                                       Faculty/student contact established

Course Progress Reports provided              Provision for student course evaluation

Comments:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           

Section 4 – Course Materials

Content equal to classroom course             Orientation of students to course achieved

Student achievement levels equal to classroom course

Explanation of use of class resources is clearly provided

Appropriate learning experiences for outcomes and challenging to students

Consistent organizational pattern                Evaluation Guidelines provided

Grading procedures are clear                       

Comments:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           

Section 5 – Community

Provides interaction for student and course material, student and faculty, and students and classmates

Personalizes learning environment              Respects talents and learning methods

Provides for timely response to questions, concerns, and comments

Includes provision for class discussion

Comments:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           

Peer Reviewer:                                                                                      Date:                                       Approved

Peer Reviewer:                                                                                      Date:                                       Approved

Peer Reviewer:                                                                                      Date:                                       Approved