Glossary

Printer-friendly version

An Investigation of the Effectiveness of Computer-Assisted Learning (CAL) for Medical and Midwifery Students
T. Wilson and G.J. Mires, University of Dundee, Scotland

Abstract
Major difficulties were encountered when we first taught intrapartum cardiotocography (CTG) through CAL in regard to hardware, network reliability, unused accounts and programme structure. After working to address these problems, a teaching session was run in which both medical and midwifery students (n=178) were given the CAL in four two-hour sessions. Students were tested on CTG interpretation before and after using the CAL programme.

The average score increased by more than 100% (p<0.0001). The enjoyment rating was > 90%. Score improvement did not correlate with computer expertise, attitude toward computers, age or enjoyment of the programme. The programme is freely available on the web and may be viewed at the University of Dundee Obstetrics and Gynecology Web site.

1. Introduction
The ability to interpret cardiotocographs (CTGs) is an essential skill for medical and midwifery students (Hull et al. 1997). Several studies have shown that teaching with computers is both possible and effective for large numbers of students in the biomedical sciences. For instance, in a study of a first year human anatomy course, students who worked with CAL before performing dissections improved the quality of their learning (Gunn and Maxwell 1996). Dewhurst et al. (1994) found that the knowledge gained from a physiology simulation of experimental work was the same as that gained in conventional lab classes. Candy et al. (1994) point out that a further advantage of CAL is that it allows self-directed education, a skill necessary for life-long learning.
Our experience with CAL has not disproved its effectiveness, but has shown that the experience can be fraught with difficulties for staff and ineffective for students. In 1999 we ran CAL sessions on CTG interpretation. The students were asked to interpret a CTG and then allowed to access a tutorial and practise sessions if they felt they needed help. The sessions were run for the third successive year, and many revisions were implemented based upon our survey findings from two previous years.

Our aims were

a)	to test the hypothesis that it is possible to use our improved CAL as an effective and enjoyable teaching tool for staff and students, and
b)	to determine if the following factors affected learning outcome: i. attitude toward computers ii. experience with computers iii. age of student iv. enjoyment of CAL

About the authors...

An external link to the University of Dundee Obstetrics and Gynecology Web site.

2. Methods
2.1 Participants and Study Design
A multiprofessional class of 143 medical students and 35 midwifery students at the University of Dundee Medical School participated in the study. The medical students were in their third year, and the midwifery students were four months into their first year of a three-year course. Students in the class were randomised to four groups because the computer suite could accommodate only 40 students. A chi-squared analysis showed that there was no significant variation between groups with regard to gender, age or student type (midwifery or medical).

Our investigation took place over three days. On the first day, all students were given a pre-test. (A post-test was scheduled immediately after the students were tutored through CAL). The protocol is described in Figure 1. The pre-test was exactly the same as the post-test, but the questions in the pre-test were of sufficient complexity that it seemed unlikely that prior knowledge of the questions would affect the scores in the post-test. An attitudes questionnaire was distributed at the same time as the post-test questionnaire. Collaboration was not permitted during testing.

The students were post-tested in four batches over two days according to the protocol shown below in Figure 1. Two days of CAL sessions were scheduled the day after the pre-test, with two groups being taught on each of the two days. The group that was taught on the second day had an additional tutorial on the topic of CTG (cardiotocography) interpretation. Tutorial sessions were used to determine if teaching by an additional method could increase the knowledge gain.

All students had a timetabled CAL session in the computer suite where 40 computers were set up. Two two-hour sessions were run on each day with three tutors in attendance: one clinician, one reproductive biochemist and one IT support person. Problems were referred to the appropriate person. (Some students who did not have accounts set up experienced logging in problems.) There were two programmes for the students to work through. The CTG programme was used last and was the only one tested in this study.

2.2 The CAL Package
In past years, interpretation of CTGs has been taught using a specially designed CAL programme, but in this study it was delivered from the Departmental Web-site for the first time. The programme was written in "Director" from Macromedia and was converted to a streamed Shockwave file for delivery from the web. Freedom of development was initially limited by the lack of soundcards and the low-resolution graphics cards, but the computers were upgraded to include sound and hi-resolution graphics towards the end of development.

Secondly, IT services upgraded the network because several students complained about the quality of the graphics. We discovered that the source of this problem was the graphics cards in the old computers that the students were using. Many students also expressed a dislike of "Text on the screen," saying they preferred to read text from a sheet of paper. In writing the program we sought to use an interrogative style that replaced text with pictures whenever possible. The students were given the opportunity for self-evaluation at intervals throughout the programme. Opportunities to explore in different directions were presented and the students could absorb information at their own pace and repeat tutorials and tests without penalty. If a question was wrongly answered the student was given the information to enable him/her to choose the correct answer.

Approximately 136h/year or 3-4 weeks was required of a full-time staff member to set up the programme.

A screenshot (22 KB) showing a summary of the objectives presented in the Cardiotocography package.

A screenshot (2 KB) of the route map from the Cardiotocography package.

An interactive demo (~540 KB) of the Cardiotocographs package.
Requires Shockwave plugin.

An external link to the authors' full-blown version of Cardiotocographs package.
Requires Shockwave plugin.

2.3 Criterion Measures
The output measures were as follows:

1. Knowledge gain. This was assessed by the difference between the pre-test and post-test scores and was measured in two ways.
   The gain is the simple difference between pre-test and post-test scores. However, because students obtaining high scores in the pre-test had little scope for improvement compared to students doing very poorly, the improvement in score was also calculated as a percentage of the potential for improvement. As the highest obtainable mark was 24, the percentage potential improvement (%PI) was calculated as
   
   

   (Post test score - Pre test score) x100 (24 - Pre test score)

   =

   %PI

   
   
2. The attitude and enjoyment of the CAL were assessed by questionnaire replies. Confounding factors, such as age of students, previous experience with computers and attitude to computers were also measured.
3. Staff were questioned as to how they enjoyed the teaching.

2.4 Evaluation and Analysis
An example CTG is shown in Figure 2.
On Day 1 of the course on labour, all students were given a pre-test (MCQ) of 12 photocopied CTGs to interpret. For each question, students were to choose from five answers:

If I saw this CTG I would be

a) very worried b) a little bit worried c) neither worried nor happy d) reassured e) I have no idea how to interpret this

Two marks were awarded for an answer that agreed with a consensus of faculty opinion. This approach was adopted because it is recognised that, even among experts, opinion might differ as to the interpretation of CAL. One mark was awarded for an answer
adjacent to the correct answer; no marks were awarded for answer e) or for more than one answer away from the correct response.

2.5 Statistical Analysis
Results were analysed by parametric statistics if Bartlett's test for homogeneity of standard deviations was insignificant. Otherwise, nonparametric statistics were used.

3. Results
3.1 Questionnaire Recovery
Of a potential 178 students, 77 % attended the first lecture of the module, and the pre-test questionnaires were completed by all present. Students were given ten minutes to complete the questionnaires, and only one person did not manage to complete it during this time. The collection of questionnaires after the CAL session was less systematic (Table 1). The computer suite allowed students to leave without being observed by staff members who were often talking to students. Some students claimed that they had urgent schedules to attend to and would return the questionnaires later, but did not do so. Therefore, although all students attending the sessions were handed a questionnaire, 18 unfilled questionnaires were collected from desks after the sessions. Estimated attendance at the CAL sessions was 69% of the class, which was similar to previous years. Some students who completed the attitudes questionnaires did not answer all the questions.

	Student numbers	% of class
Pre-test completed	137	77
Post-test completed	105	59
Both tests completed	96	54

3.2 Knowledge Gain
The mean scores of all students completing questionnaires were compared, and the data are shown in Figure 3. The improvement in scores was substantial and highly significant (p<0.0001 Student's t test). We were uncertain whether the CAL was sufficient on its own or should be combined with the human interface in a tutorial (Fig 4). Half of the students were given an additional instruction as part of an integrated clinical teaching session, and their %PI and gains were compared. The additional improvement in learning from the tutorial was not significant when measured by the %PI, but was just significant (t=2.02 p=0.0.045) when measured by the % gain.

3.2.1 Factors Contributing to Knowledge Gain
It seemed possible that there was a correlation between students' attitudes toward CAL and their gain in knowledge from CAL. When a multiple regression was performed, it was apparent that there was no relationship between %PI and enjoyment, age of students, expertise of students or attitude of students (Table 2).

The correlation matrix for %PI and enjoyment, age of students, expertise of students or attitude of students showed a significant relationship between age of students and expertise and attitude to computers and expertise (Table 3). No other factors were significant.

The correlation of expertise and attitude was highly significant (F=104.9 p>F=0.0001). The negative correlation of age and expertise was very significant (F=12.3 p>F=0.001). Although enjoyment increased with age, the relationship was not significant.

	Student’s t	p>t	Significance
Programme enjoyment	0.7	0.472	NS
Age of students	1.4	0.159	NS
Expertise of students	0.8	0.403	NS
Attitude of students	0.4	0.693	NS

	% PI	Enjoyment	Age	Expertise	Attitude
%PI	1.000	0.042	-0.180	0.144	0.074
Enjoyment	0.042	1.000	0.183	-0.007	0.069
Age	-0.180	0.183	1.000	-0.344	-0.197
Expertise	0.144	-0.007	-0.344	1.000	0.730
Attitude	0.074	0.069	-0.197	0.730	1.000

3.2.2 Attitude Measurement
Students were given a feedback questionnaire to rate their enjoyment of the programme. All students were asked

1. whether they enjoyed the CAL,
2. to rate their expertise on a 1 to 5 scale where 5 is an expert and 1 is a complete novice,
3. and to assess their liking of computers on a 1 to 5 scale where 1 indicates a strong dislike and 5 great enthusiasm.

Results are shown in Figure 5. Results from this response and others were compared to previous years' (Figure 6). There was an increase in enthusiasm for the CAL and also the level of confidence with computers. There was a small decrease in the number of students with a poor attitude towards computers.

Students were asked to rate our CAL against other CAL programmes used in their undergraduate studies. Of 87 students who answered the question, 83% considered the Obstetrics and Gynaecology CAL to be better or much better. The response rate was lower than for some other questions as some of the students claimed not to have used CAL before.

4. Discussion
As advances in information technology lead us towards more interactive, resource-based learning environments, the role of the teacher or lecturer necessarily changes from that of instructor, in the traditional sense, to that of a manager of student learning (Carter, 1997).

While CAL is effective in teaching students to manage their own learning, it is not entirely without drawbacks. Most CAL available in Dundee is text-based, unpopular with students and available for approximately half of the student modules. A major factor in developing more appealing CAL is cost; development time ascends Bloom's taxonomy as complexity increases (DLRN, 2000), and a major component of cost is staff time (Brahler et al., 1999).

The CAL presented in this paper proved a very useful manager of student learning. We found that it was an effective and enjoyable teaching tool for staff and students (Figures 3 and 5).

Knowledge gain was independent of the factors investigated. There was no significant relationship between knowledge gain and enjoyment of the programme, the age of the student, the computer expertise of the student or the attitude of the student.

4.1 Problems Encountered in the Study
4.1.1 Difficulties in Assessment
It is difficult to compare any two teaching methods (Clark 1985) such as CAL and lectures because it is hard to establish that the material and quality of teaching are comparable and that only the method differs. For this reason we decided to use a different method of assessment whereby teaching would be judged good if it met three criteria:

a) Good teaching must improve knowledge according to its aims.
b) Good teaching must be enjoyable for the students.
c) Good teaching must be enjoyable for the staff.

4.1.2 Problems with Performing the Study on Real Students
It would have been preferable to have an untaught control group against which to measure knowledge gain in the absence of the CAL. However, given the timetable constraints and the estimated probability that the students would not learn in the absence of encouragement in the form of tuition, it was decided that it would be unethical to "not teach" a group of students. Unfamiliarity with CTG could seriously impair the ability of the students to perform effectively in a labour ward, so the possibility of additional information from a control group had to be sacrificed. Also, the 20-minute tutorial sessions for groups of 40 were considered insufficient instruction for the topic.

4.1.3 Modifications Based on Prior Problems and Feedback
Although in the past, student feedback had been very encouraging, the CAL sessions had been stressful to the staff due to problems such as students not having open accounts, old equipment and unreliable network connections.

Recently, state of the art computing equipment was bought for the medical school and we were able to make use of high quality graphics and sound and a reliable high-speed network. Because the software is self-explanatory, the students asked few questions. Thus, in the most recent teaching session, the staff found they were able to relax more and generally found little to do during the CAL sessions other than supervise the students' completion of the questionnaires and chat with the students about content after the questionnaires had been completed. We felt that we were finally seeing the rewards from our long hours of preparation.

4.1.4 Problems with Questionnaire Returns
As reported, the recovery and completion of attitude questionnaires and post-tests was inconsistent; student attendance level was about 69% of the class. Information about a third of the class is, therefore, unavailable. There was no reason to suppose that the ability of the missing students differed from the attending students, but it does seem possible that students who dislike computers stayed away.

4.2 Comment on CAL's Reception
Despite the minimal input of the tutors, there were very significant gains in performance after completion of the CAL (Figure 3). However, as pointed out by Draper (1998), it is difficult to know if this was due solely to the way CAL tutored the students or whether it was an interaction between the CAL and a stimulating situation (e.g., the IT suite). By using the CAL in place of our normal teaching, we removed the uncertainty of extrapolating from a test to a real situation. We were also reassured by the low number of requests for explanation. Nonetheless, the evaluation was useful in terms of measurements in a real class situation and was very effective with little additional input from tutors.

It was interesting that additional teaching on CTG interpretation, as part of a session on labour management, improved the score as measured by the gain. However, the improvement was small (Figure 4) compared to the improvement from the CAL. We conclude that some students benefit from an additional tutorial. The subgroup that benefited has been identified as the midwifery students (Wilson and Mires 2000). Although significant, the small gain is probably not efficient in terms of effort required by the teaching staff.

Approximately 65% of the students considered themselves above average with respect to computer expertise, which contrasts with around 40% for previous years (Figure 6). In a comparable study of dental students in Dundee, the highest level of confidence in a variety of computer-based tasks was 37% (Chadwick, 1997). The improvement noted in this study may relate to increased expectation of computer use by Medical School staff.

In response to student feedback, we have been improving our CAL and the facilities for using it. This probably accounts for the observed trend in increasing enthusiasm for the programmes (Figure 6).

It is worrying that a high proportion of the students dislike computers. As attendance at the CAL sessions was not compulsory, it is possible that many of those who stayed away also dislike computers. However, we feel that the CTG CAL users overcame computer phobia in that out of six students who said they hated computers, four gave their enjoyment rating the highest level and the other two gave it the second highest rating. When asked to compare the CAL with other packages they had used, most students rated the tutorial highly (Figure 7). When constructing the package, we tried hard to make it interesting but the exact analysis of the factors contributing to its success was not addressed and will be the subject of a further study.

In conclusion, after two years of trials, we now feel the CAL is fulfilling its promise as students demonstrate significant learning increments and staff and students enjoy the teaching. The student response in this study was not related to age or computer expertise.

5. References
Ausubel D.P. (1978) In defense of advance organisers: a reply to the critics.  Review of Educational Computing Research 7 (4) 251-257.

Barnes D.J. (1999) Public forum Help seeking: the impact of providing anonymity on student help seeking behaviour.  Proceedings of Computer-Based Learning in Science Conference.  The Netherlands, July. http://www.cblis99.freeuk.com/abstract.htm#A3

Brahler C.J., Peterson N.S., Johnson E.C. (1999) Developing on-line learning materials for higher education: an overview of current issues. Educational Technology and Society 2:1-8. http://ifets.ieee.org/periodical/vol_2_99/jayne_brahler.html

Carter D.S.G. (1997) New Generation Instructional Information Technology and the Management of Teaching and Learning Paper presented at the British Educational Research Association Annual Conference. (September 11-14 1997: University of York)

Candy, P. C., Crebert G. O’Leary (1994) Developing lifelong learners through undergraduate education.  Canberra, Australia: national Board of Employment, Education and Training, Australian Government Publishing Service.

Chadwick R.G. (1997) Basic IT skills of dental undergraduates: a case for supplementary tuition at University.  Medical Teacher 19:148-149.

Clark R.E. (1985) Confounding in educational computing research. Journal of Educational Computing Research 1:137-148.

*Dearing R. (1997) National Committee of enquiry into higher education.  http://www.leeds.ac.uk/educol/ncihe/help.htm

Dewhurst D.G. Hardcastle J., Hardcastle P. T., Stuart E. (1994) Comparison of a computer simulation programme and a traditional laboratory practical class for teaching the principles of intestinal absorption.  Am J Physiol 267:S95-S104.

DLRN (Distance Learning resource Guide) 2000 Chapter 4, Bloom’s taxonomy.  http://www.dlrn.org/library/dl/guide4.html

Draper S.W. (1998) Prospects for summative evaluation.  Association for Learning Technology Journal   5:33-39.

Draper S.W., Brown M.I., Henderson F.P., McAteer E., Smith E.D. and Watt H.D. (1994) Observing and Measuring the Performance of Educational Technology. TLTP programme project.  University of Glasgow.

Gagne R.M. (1985) The Conditions of learning (4^th Ed.), New York: Holt, Rinehart, and Winston.

*Gunn C. and Maxwell L. (1996) CAL in human anatomy.  Journal of Computer Assisted Learning 12 :205-15.

*Hull M., Joyce D., Turner G., Wardle P. (1997) Undergraduate Obstetrics and Gynaecology (3^rd Edition) Oxford: Butterworth-Heinemann. ISBN 0 7506 1351 3.

Porritt N. (1997) Managing to learn with technology.  Active Learning 7:17-23.

Rothkopf E.Z. and Kaplan R. (1974) Instructional objectives to learners: effects of passage length and amount of objective-relevant content.  Journal of Educational Psychology 4:448-456.

Wilson T., Mires G.J. (1998) Teacher versus the computer for instruction: a study.  British Journal of Midwifery 6:655-658.

Wilson T., Mires G.J. (2000) A comparison of performance by medical and midwifery students in multiprofessional teaching.  Medical Education 34:744-746.

********** End of Document **********

IMEJ multimedia team member assigned to this paper	Yue-Ling Wong