Editor’s Note: Dr. Liu regularly reports his latest research through this Journal. In this instance he is validating the results of online as compared to pencil and paper evaluations for classroom and online courses.

A Comparison Study of
Online versus Traditional Student Evaluation
of Instruction

Yuliang Liu

Abstract

This comparative study was designed to investigate whether there were significant differences in student evaluation of instruction in a graduate educational research course simultaneously taught by the same instructor between an online WebCT section and a traditional section. This study used a quasi-experimental design to collect two sets of data. First, one student evaluation of instruction survey was administered online for the online WebCT section and on paper for the traditional section in an identical course at a Midwestern public university in the USA during the summer semester of 2003. Second, a student evaluation of instruction survey was also administered online for the online WebCT section and on paper for the traditional section during an identical course in the fall semester of 2004. Results from these two sets of data revealed no significant differences in student evaluation of instruction between online and traditional sections in the identical course. Further implications result from this study.

Keywords: student evaluation, online instruction, traditional instruction, quasi-experimental design, no significant difference.

Introduction

Student evaluation of instruction is widely used in most colleges and universities in the USA. Student evaluation of instruction is used by two major groups for different purposes: (a) by administrators to evaluate faculty teaching effectiveness and to make personnel decisions (e. g, tenure and promotion) and (b) by faculty to improve teaching. Typically, students are requested to complete the evaluation of instruction on paper form by using No. 2 pencils in the final week of each semester. But the format has changed recently. Presently with the rapid development of computing and Web technologies, many institutions are offering online courses. According to Waits & Lewis (2003), distance education has grown quickly in recent years. In the 2000-2001 academic year, 56% of all 2-year and 4-year institutions offered various distance education courses. In addition, 12% of all institutions planned to offer distance education courses in the next 3 years. Recent studies (Cooper, 1999; Thurmond, Wambach, Connors, & Frey, 2002) have indicated that students are satisfied with online courses. This signifies that Web-based technology is an acceptable platform for learning and instruction.

Online student evaluation of instruction has received increasing attention in recent years. In 2000, Hmieleski surveyed the 200 mostly wired institutions in the USA and found that 98% of responding institutions still used the paper-based method as the major approach to student evaluation. This indicating that online student evaluation was extremely limited outside the distance education programs. However, since Hmieleski’s study, some universities have explored other possible methods of collecting and reporting student evaluation data (Bullock, 2003; Hardy, 2003; Hoffman, 2003). Hardy reported that Northwestern University implemented a campus-wide system for online student evaluation in 1999 and found that the average numerical scores for the online evaluation were approximately the same as those for the paper-based evaluations. However, some case studies indicated that students in the online courses had a higher overall level of satisfaction. For instance, Cooper (1999) found that all students in her online anatomy course were very satisfied with her online course and the class met their expectations.

Online student evaluation has been noted as a viable alternative to the traditional paper-based method and has numerous advantages such as time efficiency, flexibility, detailed written comments, and low expense (Ballantyne, 2003; Dommeyer, Baum, Hanna, & Chapman, 2004; Sorenson & Reiner, 2003). Thus in recent years, the number of studies on online student evaluation in the literature has grown (Cantera, 2002; Hoffman, 2003; Mayer & George, 2003; McGourty, Scoles, & Thorpe, 2002). According to Hoffman (2003), although paper-based evaluation remains the major method for student evaluation data collection in traditional face-to-face courses, the use of the Internet as a primary method for collecting student evaluation data has increased approximately 8% since 2000. In addition, some institutions use both paper-based and online methods for collecting student evaluation data. At these institutions, online courses are increasingly being evaluated online and the number of face-to-face courses evaluated online has also increased.

The question of there being significant differences in student evaluation of instruction between online and paper-based methods is important since previous research indicates that the response rate in online student evaluation for the online section is likely to be lower than that in the paper-based evaluation for the traditional section. Dommeyer et al. (2004) conducted a study involving 16 instructors and undergraduate business majors. They found that students’ response rate to the online student evaluation was generally lower than that of the traditional paper-based survey. However, according to Dommeyer et al., the difference between the online evaluation and paper-based evaluation was minimal if a grade incentive was used for encouraging the online response rate. Other recent research has indicated that the response rate to the online survey can also be increased with other approaches such as the use of a sweepstakes approach (Bosnjak & Tuten, 2003; Cobanoglu & Cobanoglu, 2003).

Recent studies have indicated that the most important factor affecting student evaluation of instruction is the online learning environment. Thurmond et al. (2002) conducted a study to determine the impact of a Web-based class by controlling for student characteristics. They found that the virtual environment in the online course had a greater impact on student satisfaction than student characteristics. According to Thurmond et al., the online instructor has complete control of the virtual environment. In addition, principles of good practice (e. g, active learning and timely feedback) in traditional classrooms can also apply to the virtual classroom. This implies that if an instructor has experiences of good teaching practice in the traditional classroom, he/she will be able to transfer the good teaching practice to the virtual classroom as well. Thurmond et al.’s perspective has been supported by later studies. McGhee and Lowell (2003) compared online student evaluations in an online course with paper-based evaluations in a traditional course. McGhee and Lowell found that any possible differences in student evaluations were likely related to differences in the instructional environment.

Other comparison studies show students in online courses have similar results in their evaluation of instruction as compared to their counterparts in traditional courses. Hardy (2003) compared six courses evaluated online and on paper and found little or no overall differences in terms of the average numerical scores, the number of positive, negative, and mix comments in online and paper-based student evaluations. Hardy also found that the students who did respond wrote more detailed comments online in spite of the lower response rate. These comments provided a valuable resource for the instructor to improve teaching and learning in future course offerings. In accordance with previous studies, more recent studies have found no systematic differences between online and traditional paper-based student evaluations of instruction (e. g., Carini, Hayek, Kuh, & Ouimet, 2003; Thorpe, 2002), even when different incentives were offered to the students for the completion of online evaluations (Dommeyer et al., 2004).

The above literature review indicates that (a) online student evaluation of instruction in online courses is generally similar to that in traditional courses if both courses are taught by the same instructor, (b) the response rate in the online student evaluation in the online section is likely to be lower than that in the traditional section, and (c) students in the online section will write more detailed comments related to the course than their counterparts in the traditional section. Thus, the purpose of this study is to investigate any possible significant differences related to the above three issues in the student evaluation of instruction in the educational research course at the master level based on the course delivery method: online for the online section and paper-based for the traditional section.

The specific research hypotheses in this study are stated as follows:

Hypothesis 1: There was no statistically significant difference in student evaluation in a graduate educational research course between the online section and the traditional section if both sections were taught by the same instructor in the same semester.

Hypothesis 2: The response rate in the online student evaluation in the online section was lower than that in the traditional section.

Hypothesis 3: Students in the online section wrote more detailed comments related to the course than their counterparts in the traditional section.
 

Method

Participants

The participants in this study were recruited based on convenience sampling on two occasions. The first set of data collected for this study was in the summer semester of 2003. In that semester, the author was assigned to simultaneously teach one online section and one traditional section of the educational research course. All students who self-selected to enroll in this course for 10 weeks during the summer semester of 2003 were solicited in the first week of the semester for participation in this study. The educational research course is a required core course in education at the master’s level at a Midwestern state university in the United States. Students in this course were from different graduate programs in education. Twenty-four students enrolled in the online section, but two of them withdrew within the first two weeks due to time commitment and unexpected family issues. Twenty-two students in the online section were included for final analysis and twenty-one students enrolled in the traditional section. Thus, a total of 43 participants in both sections were recruited to participate in the study. Participants in both sections were asked to complete consent forms and demographic surveys in the first week. A pretest of course content in both sections was administered. A preliminary analysis of the pretest revealed that although the traditional section scored a little higher than the online section, but no significant differences were detected between the two sections.

The second set of data collected for this study was in the fall semester of 2004. In that semester, the author was assigned again to simultaneously teach one online section and one traditional section of the same above course. Similarly, all students who self-selected to enroll in this course for 16 weeks during that semester were solicited in the first week for participation in this study. Students in this course were from different graduate programs in education. Nineteen students enrolled in the online section and twenty-one enrolled in the traditional section. Thus, a total of 40 participants in both sections were recruited to participate in the study. Participants in both sections were asked to complete consent forms and demographic surveys in the first week. A pretest of course content in both sections was administered. A preliminary analysis of the pretest revealed that no significant differences were detected between the two sections.

Instruments

Although many researchers (e. g., Harrington & Reasons, 2005) have recently proposed developing useful online student evaluation of instruction for distance education courses, this study used the existing student evaluation forms of instruction from the author’s department. The author’s department decided to use the same student evaluation of instruction for both traditional and online courses. That is, online courses were evaluated online via WebCT and traditional courses were evaluated using the paper-based approach in the traditional classroom during the last two weeks of each semester. However, both traditional and online courses used the same student evaluation survey. Thus the student evaluation survey in both types of classes has the same validity and reliability. The administration of the student evaluation was anonymous and confidential. During administration, the instructor was required to leave the classroom for the traditional sections. A student volunteer was asked to seal the completed evaluation surveys in the envelope and to take the envelope back to the department’s secretary. In the online section on WebCT, students evaluation results were not related to their identifications such as names.

The student evaluation survey used in the summer semester of 2003 had 16 five-point Likert scale items: 1‑Poor, 2‑Fair, 3‑Average, 4‑very Good, and 5‑Superior. There was one additional item to evaluate the level of difficulty of the course, from Very Easy (1) to Very Difficult (5). Students circled the appropriate number for each item. In addition to these numeric items, students could also write comments.

Since the fall semester of 2003, the author’s department revised the student evaluation of instruction survey and approved the new version. The new student evaluation survey used in the fall semester of 2004 had 18 five-point Likert scale items: from 1-Strongly Disagree to 5‑Strongly Agree. Similarly, there was one additional item to evaluate the level of difficulty of the course, from Very Easy (1) to Very Difficult (5). Students circled the appropriate number for each item. In addition to these numeric items, students were provided with a space to write open comments.

After the administration of each student evaluation survey, the departmental secretary added each student’s evaluation result, calculated the average score for each item, and the average score for all 16 (in 2003) and 18 (in 2004) items for each section. Students’ qualitative comments were typed and added to the report as well. The results were then released to the administrators and the faculty.
 

Research Design

This study used a non-equivalent control group design. In both the experimental group (online via WebCT) and control group (traditional classroom), the dependent variables of learning performance were pretested and posttested. The dependent variable of student evaluation of instruction was completed in the final two weeks of the semester, online for the online section and paper-based in the traditional section. The independent variable was online vs. traditional instruction in a graduate course. Based on recommendations from the Institute for Higher Education Policy (2000) and Kearsley (2000), a hybrid of instructional techniques were employed in the online section. Specifically, several major features of WebCT were used throughout the semester such as weekly online writing, peer critiquing, bulletin board discussion, online testing, and e-mail. Constructivist learning theory was the major theoretical foundation for online instruction in this course. Instructional design was based on the ADDIE model (Analysis, Design, Development, Implementation, and Evaluation) proposed by Dick, Carey, and Carey (2001). For additional information on design, development and instructional strategies used in this course, see other recent publications by the author (Liu, 2003a; 2003b).

To reduce learner anxiety and maximize learning, one Face-to-Face (FtF) orientation was conducted during the first week for the online section. The traditional section met once a week for 3 hours, and was primarily taught FtF throughout the semester. Both sections were taught simultaneously by the lead investigator in the summer semester of 2003. In order to make the sections as equivalent as possible, the instructional objectives, content, requirements, assignments, and assessments in both sections were the same.

Procedure

The pretest was administered in paper-and-pencil format to both sections during the first week of the semester to determine initial learning and performance. The participants in the online section were introduced to the online WebCT environment from the second week through the final week. Ongoing posttests, including chapter quizzes, a final test, and student evaluation of instruction were administered online for the online section and administered in paper-and-pencil format for the traditional section.
 

Results and Discussion

Pretests and posttests of learning performance, as well as student evaluation data in both sections in the summer semester of 2003 and in the fall semester of 2004 were coded and analyzed using SPSS 12.0. Regarding students’ learning outcomes in the summer semester of 2003, there was a significant difference in most chapter quizzes and the final test between online and traditional sections. Specifically, online learners outperformed their counterparts in the traditional section (Liu, 2005a). In addition, regarding students’ learning outcomes in fall 2004, there was not a significant difference between online and traditional sections. That is, online learners performed as well as their counterparts in the traditional section (Liu, 2005b).

Research Hypothesis 1

Regarding students’ perceptions and satisfactions with the course, the student evaluation survey used in the summer semester of 2003 found that the average scores and standard deviations (SD) for all 16 items combined for online and traditional sections were, respectively, 4.5 with a SD = .23 and 4.3, with a SD of .34. The descriptive statistics for all the items in this survey are presented in Table 1. Results from the paired t test in Table 2 revealed that a significant difference in student evaluation of instruction between online and traditional sections was only detected in item 15 (t = 2.08, p = .044), but no significant differences were found in the other 15 items (p > .05). Item 15 asked students to give an overall rating of this instructor's general teaching effectiveness (related to the course objectives and new understanding). The results of item 15 showed that the online section gave a higher rating (mean = 4.77, SD = .43) while the traditional section gave a lower rating (mean = 4.29, SD = 1.01).

The student evaluation survey used in the fall semester of 2004 found that the average scores and standard deviations (SD) for all 18 items for online and traditional sections were, respectively, 4.4 with a SD = 1.1 and 4.4, with a SD of 1.0. The descriptive statistics for all the items in this second survey are presented in Table 3. Results from the paired t test in Table 4 revealed that no significant differences were found in all 18 items (p > .05). In addition, the major reason for the large SD in both sections in the fall semester of 2004 is that there was one student (outlier) who misunderstood the survey instructions and completely chose “1” for all 18 items in each section. This can be verified from that student’s very positive qualitative comments.

Table 1

Descriptive Statistics for Summer 2003 Student Evaluation Items

Table 2

Independent Samples t Test Results for Summer 2003 Student Evaluation Items

 In addition, the last additional item from both sets of data described previously found that a majority of students in both online and traditional sections indicated this course as either “Moderately Difficult” or “Very Difficult” among the five options. The numeric averages in two online sections and two traditional sections were all between 4 (Moderately Difficult) and 5 (Very Difficult) (see the last item in Tables 1 and 3). This result was not surprising for the author of this study since this is a graduate course that requires rigorous instruction. In addition, this seems to be one of the most difficult and challenging course in all educational graduate programs. Results in this study showed that the research hypothesis 1 was supported. This is consistent with findings in other studies. Recent studies have consistently found no systematic differences between online and traditional paper-based student evaluations of instruction (e. g., Carini, et al., 2003; Hardy, 2003; Thorpe, 2002), even when different incentives such as grade were offered to the students for the completion of online evaluations (Dommeyer et al., 2004).

Table 3

Descriptive Statistics for the Fall 2004 Student Evaluation Items

Table 4

Independent Samples t Test Results for Fall 2004 Student Evaluation Items