IEEE ©2001 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.

Return to Table of Contents


An Evaluation Model for Web-Based Instruction

Woochun Jun and Le Gruenwald, Member, IEEE


Abstract - Recently Web-Based Instruction (WBI) has been adopted for many educational systems in order to support distance education. WBI has become popular in that it overcomes time and space limitation in traditional educational systems. But due to lack of face-to-face communication, it is crucial that WBI provide interactivity and motivation for students. This paper introduces a formal model that evaluates interactivity and motivation for courses based on WBI. The model is comprehensive and objective so that it can be used to evaluate any course. Based on the model, the paper selects some WBI courses and compares them for their interactivity and motivation. Finally, the paper concludes with a discussion of further research issues.


I. Introduction

Recently, the growth in Internet technology usage has affected the traditional education methodology. Especially, since the World Wide Web (WWW) has emerged to become a global, interactive, dynamic medium for sharing information all around the world. Due to its ability to deal with multimedia information, such as graphics, sound, and video, and easy-to-use web browsers, it can be useful for many educational systems. WBI is an informal instructional system using WWW services and other technical supports, such as chat-rooms and email. WBI is more formally defined as a hypermedia-based instructional program, which utilizes the attributes and resources of the WWW to create a meaningful learning environment where learning is fostered and supported [1].

In order to constitute an educational system, WBI has the following components [1]: content developments, such as instructional design and curriculum developments, multimedia components, Internet tools, computers and storage devices, connection and service providers, authoring programs, such as HTML and JAVA, servers, browsers, and other applications. WBI features are characterized by those components and are classified into two categories: key features and additional features. Key features are an integral part of the WBI design, such as interactive and multimedia systems. On the other hand, additional features are secondary tools assisting WBI designs, such as ease of coursework development.

WBI has changed the traditional relationships between teachers and students. In the traditional instruction setting, teachers and students share the same space at the same time and also, students may work individually or in groups. On the other hand, in WBI, teachers and students may meet asynchronously using communication tools, such as video and sound, at different places. Also, WBI can encourage students to work in a self-driven manner.

Although WBI has affected the traditional education system tremendously, it still has some limitations. The lack of face-to-face communication is a typical example. Thus, the major concern in WBI is how teachers can motivate unseen students. There are various ways to provide motivation and interactivity in WBI, but their approaches are somewhat diverse and subjective. This paper provides a formal model that evaluates interactivity and motivation for courses based on WBI. It is objective and based on principles dominating pedagogical philosophy and learning theory.

This paper is organized as follows. In Section II, related works providing motivation and interactivity are presented. In Section III, a formal evaluation model is introduced. Based on this model, some representative WBI courses are selected and compared for their interactivity and motivation in Section IV. Finally, conclusions and further research issues are discussed in Section V.


II. Related Work

In [2], the authors argue that course design and the teacher's role might affect student motivation. Based on these arguments, they present three ways to enhance the use of WBI: how to motivate students, course design considerations, and how to motivate instructors.

First, the causes for lack of student motivation include lack of preparedness, lack of funds to purchase electronic resources, initial difficulty in using advanced hardware and software, phobia about technology, lack of background on course, interpersonal difficulty, family illness, etc. The authors suggested the following solutions: get to know the student by providing personal information, such as biographies, photographs, and e-mail addresses; avoid confrontations, sarcasms, or put-downs; if a face-to-face meeting cannot be arranged, use other methods (such as e-mail) to communicate.

Second, as possible course design considerations for providing motivation, the following principles may apply: variation and curiosity, relevance, challenge level, positive outcomes, positive impression, readable style, and early interest. Variation and curiosity refer to providing diversity and making changes in content to stimulate attention and curiosity. Relevance refers to connecting student learning to objectives of the course. Challenge level refers to providing continuing interactions. Positive outcomes refer to providing rewards or the opportunity to present work to other students. Positive impression refers to providing a good impression by arranging graphics, text, maps, etc. Readable style refers to providing readable messages and dialogues. Early interest refers to stimulating early interest in a course.

Finally, the authors argued that the motivation problem may also affect teachers, and unmotivated teachers may impact the entire class. In order to motivate the instructors themselves, the authors provided suggestions, such as take more time to redesign the course and develop the ability to deal with change.

In [3], the authors suggested three ways of interaction: student-course material interaction, student-teacher interaction, and student-student interaction. First, for student-course material interaction, a course should be designed to facilitate self-directed learning, and include many illustrations and guidelines. Second, for student-teacher interaction, teachers need to constantly motivate students. Finally, in order to support student-student interaction, communication tools should be provided.

In [4], the authors classified web evaluation criteria into six categories: a) content, b) multimedia components, which are graphics, sound and video, c) authority, which includes the quality of the web page writer, and copyright and trustworthiness of web management organization, d) currency, which refers to whether the web pages' content is up-to-date or not, e) general appearance, which refers to attractiveness of the web page, and f) ease of navigation.

In [5], a way to provide motivation is presented. It is discussed in terms of four categories: attention, relevance, confidence and satisfaction. Attention represents interests or curiosities of students. Thus, students are supposed to answer yes on questions like "Does the display provide curiosity?" or "Does the content invoke curiosity?". Relevance is the applicability of the content to real life or other subjects. Confidence is to describe the students level of understanding of the course material. Finally, satisfaction is to provide the students' satisfaction after the course, including fairness in grading, positive rewards and psychological impacts.

The related works above have the following problems in order to evaluate interactivity and/or motivation. The works in [2,5] are a comprehensive guide for providing motivation in WBI, but include psychological effects that are subjective and are not easily measurable. The work in [3] establishes the only general rules for providing interactivity, which may not be useful for WBI designers directly.Finally, the web evaluation criteria in [4] include so many subjective elements that they may lose generality.


III. The Proposed Model

Based on the related works in Section II, a formal model needs to be developed for WBI evaluation criteria. The model should be objective so that it can be useful for all WBI designers and teachers. In this section, an evaluation model for interactivity and motivation in WBI is presented. Only objective elements are selected so that the model can be used as guidelines for future WBI designers. Note that, in order not to loose generality, these elements are based on the major pedagogical philosophy and learning theory, which will be discussed later. For the evaluation of interactivity and motivation, interactivity and motivation are classified into three categories: student-to-course content relationship, student-to-teacher relationship, and student-to-student relationship. The evaluation criteria for interactivity include any elements inducing student reaction. On the other hand, the evaluation criteria for motivation include any elements stimulating student interest. The elements for interactivity and motivation are somewhat overlapping. The evaluation model may not be an absolute index for WBI products, but it can be used for comparing the relative performance of any web pages.

The following two educational guidelines are adopted for interactivity and motivation: learning theory and pedagogical philosophy. Although there are many learning theories, two theories, behaviorism and cognitivism, dominate instructional design. Each theory is summarized as follows [1].

Behaviorism emphasizes interactive learning systems. In fact, behaviorism considers the critical factors in learning observable behavior and instruction, which involves shaping desirable behaviors through stimuli, responses, feedback, and reinforcement. Thus, once a stimulus, which is in the form of a short presentation of some content, is provided, a response is expected. Feedback is provided as to the accuracy of the response. Also, positive reinforcement is given for accurate responses. For inaccurate responses, additional or modified stimulus is given so that another cycle begins.

On the other hand, cognitivism places more emphasis on internal mental status than on behavior. These internal mental states include simple propositions, schema, general rules, skills, general skills, automatic skills, and mental models [7]. According to cognitivism, in order for some type of knowledge to be constructed by the learner, various learning strategies, such as memorization, direct instruction, deduction, drill and practice and induction, are required.

The pedagogical philosophy ranges from a strict instructivism to a radical constructivism, as follows. Instructivism emphasizes the importance of objectives existing apart from the learners. That is, once objectives are identified, they are sequenced into learning hierarchies. Those hierarchies usually represent a progression from lower to higher order learning, and direct instruction is provided to address each of the objectives in sequence. In instructivism, learners are considered as passive recipients of instruction. It is also based on objectivist epistemology that defines knowledge as separate from knowing. That is, reality exists regardless of the existence of sentient beings. Also, humans acquire knowledge on this reality in an objective way through senses.

On the other hand, constructivism emphasizes the learner's intentions, experience, and cognitive strategies. According to constructivism, learners construct different cognitive structures based on their previous knowledge and what they experience in different learning environments. Thus, constructivists are supposed to have learning environments that are as rich and diverse as possible. Unlike instructivism, each learner is regarded as an individual replete with pre-existing knowledge, aptitudes, motivations, and other characteristics that are usually difficult to assess. Thus, direct instruction is replaced with tasks to be accomplished or problems to be solved that have personal relevance for learners. Constructivists also believe that knowledge does not exist outside the minds of human beings and that what we know of reality is individually and socially constructed based on the learner's previous experience.Also, they believe that learning consists of acquiring viable strategies that meet one's objectives, and learning can be estimated only through observation and dialogue.

Each of the evaluation models for interactive and motivation consists of three relationships: Student-to-Course Content, Student-to-Teacher, and Student-to-Student. Within each relationship, a number of elements that may affect the relationship are identified.Each element is given a score of 1 if it exists and 0 otherwise.Formulas based on the scores given to these elements for all three relationships are derived to evaluate interactivity and motivation.Below are the detailed evaluation models, where SCI1...SCI5 represent the scores for the five elements affecting the Student-to-Course Interactivity, STI1...STI5 represent the scores for the five elements affecting the Student-to-Teacher Interactivity, and SSI1...SSI3 represent the scores for the three elements affecting the Student-to-Student Interactivity.Similarly SCM1...SCM14, STM1...STM4, and SSM1...SSM5 are for motivation.

1.Interactivity

1) Student-to-course content relationship

2) Student-to-teacher relationship

3) Student-to-Student relationship

2.Motivation

1) Student-to-course content relationship

2) Student-to-teacher relationship

3) Student-to-student relationship

For interactivity, the degree of overall interactivity, I, is determined by the following expression. We assume that, the bigger the value I, the higher the achieved interactivity for a specific WBI course. This means that, no matter what educational theory each element comes from, we simply assume that the element, with possibly different weight, is helpful for achieving overall interactivity. The value of I can be somewhere between 0 and 1.

I = C1 * SCI + C2 * STI + C3 * SSI--- (1)
Where 0 <= I <= 1
C1 + C2 + C3 = 1---(2)
Where 0 <= C1 <= 1, 0 <= C2 <= 1, 0 <= C3 <= 1

Where C1, C2 and C3 are coefficients (or weight value), and SCI (the degree of student-to-course interactivity), STI (the degree of student-to-teacher interactivity) and SSI (the degree of student-to-student interactivity) are determined as follows. We assume that the degree of each type of interactivity has higher value as more elements are provided for the type.

SCI = (SCI1 + SCI2 + SCI3 + SCI4 + SCI5)/5---(3)
STI = (STI1 + STI2+ STI3 + STI4 + STI5)/5---(4)
SSI = (SSI1 + SSI2 + SSI3)/3---(5)
Where 0 <= SCI <= 1, 0 <= STI <= 1, 0 <= SSI <= 1

Similarly, for motivation, the degree of overall motivation, M, is determined by the following expression. As in interactivity, it is assumed that, the higher the value of M, the higher the achieved motivation for a WBI course. In turn, this means that, no matter what educational theory each element comes from, we simply assume that the element, with possibly different weight, is helpful for achieving overall motivation. Also, the value of M can be somewhere between 0 and 1.

M = H1 * SCM + H2 * STM + H3 * SSM ---(6)
Where 0 <= M <= 1
H1 + H2 + H3 = 1---(7)
Where 0 <= H1 <= 1, 0 <= H2 <= 1, 0 <= H3 <= 1

Where H1, H2 and H3 are coefficients (or weights), and SCM (the degree of student-to-course motivation), STM (the degree of student-to-teacher motivation) and SSM (the degree of student-to-student motivation) are determined as follows.We assume that the degree of each type of motivation has higher value as more elements are provided for each type.

SCM = (SCM1 + SCM2 + .... + SCM14)/14---(8)
STM = (STM1 + STM2 + STM3 + STM4)/4---(9)
SSM = (SSM1 + SSM2+ SSM3 + SSM4 + SSM5)/5---(10)
Where 0 <= SCM <= 1, 0 <= STM <= 1, 0 <= SSM <= 1


IV. Case Studies

In this Section, some representative educational web pages are selected and they are compared for their interactivity and motivation provided. For ease of comparison, all web pages are selected from science-related subjects. Those three web pages are as follows.

  1. Title: Weather Watchers (Course1)
    This web page gives weather information on some countries and also introduces the lifestyle of those countries according to weather.
    Web Address: http://163.150.1.6/curriculum/weather.html
  2. Title: Is There Water on The Moon (Course2)
    This web page provides information on what to do with the possible discovery of water on the lunar surface.
    Web address: http://207.180.0.32/~fasulo/water.htm
  3. Title: Animals of The Tropical Rain Forest (Course3)
    This web page gives ideas on animals of the tropical rain forest. Those ideas include common and scientific names of animals, the animals' habitat and food, descriptive details or pictures of the animals, protection behavior of the animals.
    Web address: http://www.plainfield.k12.in.us/hschool/webq/webq3/rain.htm

We first assume that C1 = C2 = C3 = 1/3. That is, we assume that each type of interactivity is equally important. According to a specific WBI course, C1, C2 and C3 may have different values. Using formulas (3), (4) and (5) in the previous section, Fig. 1 shows the values of interactivity in terms of SCI, STI and SSI for three courses, Course1, Course2 and Course3. All three courses have the same value for interactivity. For example, for SCI, all three courses provide hyperlinks, scrolling and multimedia data, but not question/answer guides and exercises. Similarly, for STI, all three courses provide communication media and collaboration between students and a teacher, but not virtual office hours, rewards for accomplishments and extra contact information. The student-to-content interactivity has a higher value than the student-to-teacher interactivity and student-to-student interactivity. This means that more student-to-teacher and student-to-student relationships should be taken into consideration. Using formulas (1) and (2), Fig. 2 shows the overall interactivity for the three courses.

As in the case of interactivity, we also assume that H1 = H2 = H3 = 1/3. For simplicity, we assume that the importance of each type of motivation is equal. Using formulas (8), (9) and (10), Fig. 3 provides the comparison of motivation for each SCM, STM and SSM. From Fig.3, we know that Course2 provides the best motivation for SCM while Course1 provides the best motivation for SSM. For STM, all three courses have the same values. As in interactivity, student-to-teacher and student-to-student relationship should be emphasized for the future WBI designs. Using formulas (6) and (7), Fig. 4 shows the comparison of overall motivation. Course2 and 3 shows the best value in motivation.


V. Conclusions and Future Research

In this paper, we present a formal model that evaluates interactivity and motivation in WBI. The proposed model includes comprehensive and objective elements in WBI. The model classifies interactivity and motivation into three categories: course-to-student relationship, teacher-to-student relationship and student-to-student relationship. Based on the model, three educational web sites are selected and compared for the their interactivity and motivation. This model can be used when establishing guidelines for educational web page design.

We are developing a comprehensive evaluation model in which some psychological effects are added. Also, in addition to interactivity and motivation, we are developing an analytical model that can be used to test various aspects of WBI products.


References

[1]B. H. Khan, "Web-based Instruction (WBI): What Is It and Why Is It?" in Web-Based Instruction, edited by B. H. Khan, Educational Technology Publications, New Jersey, USA, 1997.

[2]P. Duchastel, "A Motivational Framework for Web-based Instruction", in Web-Based Instruction, edited by B. H. Khan, Educational Technology Publications, New Jersey, USA, 1997.

[3]E. Cho, "Distance Instruction", The Journal of KIPS, Vol. 4, No. 3, May 1997, pp. 20 ? 28.

[4]Y. Son and K. Kim, "Evaluating Instructional Web Pages with Web Evaluation Model", Proceedings of the 4th KAIE Winter Conference, 1999, pp. 320 - 328.

[5]S. Song, Providing Motivation in Web-Based Instruction, in Web-Based Instruction, edited I. Na, Educational Science Press, May 1999, Seoul, Korea

[6]B. Willis, Distance Education: Strategies and Tools, Englewood Cliffs, Educational Technology Pub., 1995.

[7]P. Kyllonen and V. Shute, "A taxonomy of learning skills", in Learning and individual differences: Advances in theory and research, edited by P. Ackerman, R. Sternberg, and R. Glaser, W. H. Freeman and Company, New York, 1989.


Author Contact Information

Woochun Jun
Dept. of Computer Education
Seoul National University of Education
Seocho Dong, Seocho Gu
Seoul, Korea
Phone: 82-2-3475-2504
Fax: 82-2-3475-2263
E-mail: wocjun@ns.seoul-e.ac.kr

Le Gruenwald
School of Computer Science
The University of Oklahoma
200 Felgar Street, Room 114 EL
Norman, OK 73019
U.S.A.
Phone: 1-405-325-3498
Fax:1-405-325-4044
E-mail: ggruenwald@ou.edu


Author Bibliographies

Woochun Jun has been an assistant professor in Dept. of Computer Education at Seoul National University of Education, Seoul, Korea since Aug. 1998. Currently, he teaches information communication, database design, internet resource management, etc. His areas of interest include web-based instruction, computer-based instruction, transaction processing and concurrency control in real-time databases, object-oriented databases, and distributed databases. He holds a Ph. D degree in Computer Science from the University of Oklahoma, USA, in 1997. He also got a Masters degree and BS degree in Computer Science from Sogang University, Seoul, Korea, in 1987 and 1985, respectively.

Dr. Le Gruenwald is a Presidential Professor and an Associate Professor in the School of Computer Science at The University of Oklahoma.She received her Ph.D. in Computer Science from Southern Methodist University in 1990, MS in Computer Science from the University of Houston in 1983, and BS in Physics from the University of Saigon, Vietnam in 1978.She was a Software Engineer at WRT, a Lecturer in the Computer Science and Engineering Department at Southern Methodist University, and a Member of Technical Staff in the Database Management Group at the Advanced Switching Laboratory of NEC America.Her major research and teaching interests include Distributed Mobile Databases, Real-Time Databases, Object-Oriented Databases, Data Warehouse and Data Mining, Multimedia Databases, and Web Databases.She is a member of ACM, SIGMOD, and IEEE Computer Society.


Return to Table of Contents