EVALUATING QUALITY IN FULLY ONLINE U.S. UNIVERSITY COURSES: A COMPARISON OF UNIVERSITY OF MARYLAND UNIVERSITY COLLEGE AND TROY UNIVERSITY

A key component of quality assurance in online higher education is the periodic evaluation of fully online courses, by both internal and external reviewers, against standards developed by the offering institution. These standards can address a variety of quality areas including but not limited to: the organization and structure of the online course; the extent to which technology is used to foster learning and student engagement; and the use of available communication features to stimulate student discussion and interaction. In this paper the online evaluation processes and criteria of the two largest U.S. state universities involved in online education – University of Maryland University College and Troy University – are compared. It will be shown that the two institutions arrived independently at very similar quality standards for online courses, and that these standards are congruent with those developed and promulgated by nationwide higher education accreditation agencies in the U.S.


Introduction
Over the past decade the number and popularity of fully online U.S. university courses and degree programs have risen sharply.University of Maryland University College (UMUC), which has one of the largest online academic initiatives in the world, saw its online enrollments grow from slightly more than 20,000 in 1999 to more than 143,000 in 2005; an astonishing 615% increase in six years.The technologies that underlie the Internet and World Wide Web have elevated "distance education" from what used to be a somewhat marginal position on U.S. university campuses to one of the most significant current issues in U.S. higher education.
The measurement and demonstration of quality is a sine qua non for online higher education, the more so because this model is new and very different from the traditional "face-to-face" classroom delivery of courses.In the U.S., higher education policy, including federal government programs for student financial aid, is still grounded in a traditional approach based on young, full-time, largely residential students who accomplish the majority of their degree coursework face-to-face, on campus and during a four-or five-year period of time.This approach is incongruent with the new economic and workforce realities in the U.S., where working adults must attain and constantly update large and continually changing bodies of knowledge in order to remain competitive.As the need for "lifelong learning" opportunities for adults grows, the demand for college and university coursework offered in a form convenient for such students grows with it, and online asynchronous delivery is currently the most powerful tool for providing courses, programs, and services to such students.In order to transition online learning into the U.S. policy mainstream, institutions have had to develop ways of measuring and demonstrating the quality of their online offerings.This paper examines one type of institutional quality assurance practice, the use of a rubric to evaluate the content and activity of an online course, in each of two U.S. institutions, UMUC and Troy University (TROY).

National Quality Standards
National standards for quality in U.S. online higher education grew out of earlier projects dealing more generally with technology-or telecommunicationsmediated education.One of the most influential of these projects, carried out by a group of representatives from the higher education oversight bodies in several Western U.S. states, led to the publication in 1995 of a set of principles for measuring quality in electronically-delivered courses and programs.The current version of these principles was published by the Western Cooperative for Educational Telecommunications (WCET) in 2005 as Principles of Good Practice for Electronically Offered Academic Degree and Certificate Programs (WCET, 2005).While broad, these principles cover institutional and course components such as: Building upon the WCET principles, the Council of Regional Accrediting Commissions (CRAC), an umbrella organization of the eight U.S. regional accrediting agencies, issued in 2005 its own Best Practices for Electronically Offered Degree and Certificate Programs (CRAC, 2005).Like the WCET principles, the CRAC best practices cover the role of the institution in assuring quality not only of course content and communication but also of the student and faculty support services necessary to ensure a high quality learning experience.Both sets of standards also call for institutions to evaluate and assess student learning outcomes in online courses with the goal of continuous quality improvement.

Institutional Quality Standards: Two Examples
UMUC, an accredited state university in Maryland, has been an adult and continuing higher education institution since its inception in 1947.In pursuit of its mission UMUC has experimented with many non-traditional delivery models and technologies.Since the mid 1990s, however, fully online asynchronous delivery through the use of its proprietary learning management system WebTycho has emerged as the most effective delivery model.TROY, which like UMUC is a state institution (based in Alabama), also serves adult and continuing students throughout the world using a variety of delivery formats and techniques.TROY makes use of the commercial learning management system Blackboard® to provide its array of fully online courses and programs.
Both institutions independently created evaluation standards and checklists for online course quality.Both use a rubric in order to provide evaluators with the institutions' expectations and standards for quality.Although the rubrics and the documents in which they are embedded differ in terms of structure, they are surprisingly uniform in terms of the areas of importance and the means by which minimum levels of quality are displayed.

Troy University
TROY created a comprehensive Course Evaluation Tool (Troy University, 2004) for use by reviewers to evaluate the quality of its online courses.The TROY Course Evaluation Tool uses a rubric divided into five sections: In each of the five sections reviewers are provided with areas for the application of best practices for quality along with examples (including screen captures from exemplary classes) of high quality usage of the various features.Table 1 shows a summary of these best practice areas: For each of the best practice areas the evaluator is asked to provide a rating on a Likert scale of 1 (usually the lowest level of quality) to 5 (usually the highest level of quality).Thus, for the item "the instructor's use of the discussion board" the evaluator provides one of the following ratings: A cumulative score for all five sections of the rubric provides TROY faculty and staff with a baseline assessment of the quality of a particular online course as well as guidance for improving those specific sections that need additional attention in order to achieve minimum levels of quality.The TROY e-Campus will conduct assessments of every course on a rotating basis to ensure that every course is assessed in 18 month increments.

University of Maryland University College
UMUC also created a rubric-based evaluation tool, Online Classroom Observation Form, for the assessment of its online courses.The UMUC rubric is framed around specific functional areas within the online classroom: 1  UMUC's evaluation form uses a "Yes/No/Not Applicable" rating in response to quality statements such as "main topics [within a threaded conference] crafted to foster student-student interaction."The form also provides space for the evaluator to give more detailed narrative feedback on the quality of the various class components.Interaction is a key quality factor in all functional areas covered by the form.
UMUC also provides its online faculty with a separate document entitled Expectations of Classroom Setup and Online Teaching (UMUC, n.d.).This document gives faculty the guidelines and expectations that are measured by the observation form.In this document, too, there is a great emphasis on the need for interaction in general and instructor feedback to students in particular.
At present UMUC conducts internal reviews of online courses at least every two years, more frequently for courses that deal with rapidly changing technical topics.In UMUC's School of Undergraduate Studies, faculty members designated as "course chairs" review every online section of every course prior to the start of a semester.In addition, academic departments conduct full reviews of course sections (using the Online Classroom Observation Form) for purposes of faculty evaluation and promotion.

The Role of Interaction
A key quality factor in both the WCET and CRAC standards is interaction between the instructor and students, and among the students themselves, in electronically-delivered courses.A review of Tables 1 and 2 above show that TROY and UMUC both place great emphasis on interaction in their quality standards for online courses.Studies of student and faculty experiences in and perceptions of online courses have confirmed the importance of interaction as both a source and a measure of quality.Wang, Newlin, & Tucker (2001) found a correlation between the number of student communications in an online coursespecifically, responses to problems or examples given by the instructor -and the students' final course grades.Beyond the basic academic and informational content of the communication in online courses, Roblyer & Wiencke (2003) have commented on the fact that students in online courses typically "exchange messages and form perceptions of each other, of the subject matter content, and of the course; these exchanges and perceptions affect the nature of messages and thus of the learning processes that take place" (p.80).Other studies suggest that interaction is the main criterion by which both students and faculty judge the quality of an online course.For example, Lao and Gonzales (2005) found widespread agreement among faculty members about the importance of feedback to students in an online environment, despite the fact that they also agreed online teaching was more time-consuming than face-to-face.Anderson and Puckett (2005) noted that in-service teachers who completed the practical, field experience component of a graduate-level literacy course fully online also made use of online interaction tools to create a "learning community" and an enriched inventory of resources and perspectives of others.
These and other studies and commentaries (e.g.Jin, 2005;Moore & Marra, 2005;Hosie & Schibeci, 2005;Yu, Durrington, & Olinzock, 2005) confirm the central role of interaction in the design and delivery of a high quality online course.They also point to the benefit of the use of rubrics to assess quality, especially in institutions with a large number and wide range of online courses.

Conclusion
Although U.S. quality standards for online higher education arose separately in accrediting agencies and individual institutions, they nonetheless have a number of factors in common.Institutional commitment to online education, especially as measured by the provision of student and faculty services and continual assessment of quality and effectiveness, seems to be a basic component of all sets of standards.Effective use of the various communication options for each type of delivery, in order to maximize interaction within the electronic classroom, has also emerged as an important factor.Quality standards, like the delivery technologies and models themselves, will doubtless continue to evolve over the next decade.The pedagogical primacy of interaction, however, is likely to remain regardless of the rate or nature of future technological changes.
1. Course Organization and Structure 2. Interactivity and Communication 3. Course Content 4. Usability and Scalability 5. General Instructional Design