Developing
a Strategy for Creating and Assessing Digital Media Curriculum Material
Jennifer
Burg, Wake Forest University
Yue-Ling Wong, Wake Forest
University
Leah McCoy, Wake Forest University
Abstract
Creating curriculum
material in digital media is a challenge, not only because of its many facets,
but also because of the tug-of-war between application-oriented instruction
versus fundamental concepts. Our work is an attempt to respond to the challenges
of digital media curriculum development by dividing the material into three
modules with parallel chapters: a primer, an art module, and a computer science
module. An additional component of our digital media curriculum development
project is an assessment of the pedagogical value of the instructional modules
we have created. One of the special challenges in assessing digital media curriculum
material is making the connection between concepts and applications. The two
traditional methods of assessing student performance -- evaluating either their
final projects or their mastery of concepts through tests and quizzes -- fail
to associate "knowing" with "doing." This paper describes
an assessment experiment recently conducted to help us gain insight into students'
thought processes. Students are observed and videotaped in a closed lab designed
to help them understand concepts and to help us assess how they relate concepts
to activities. Based on our observations, we propose a refinement to our assessment
procedures and improvements to our curriculum material.
1. Background
Digital media is an inherently
interdisciplinary area of study with links to computer science, art, film,
music, communications, architecture, and design [Burg, Wong, and Strokanova
2004]. Creating appropriately targeted curriculum material in digital media
is a challenge, not only because of its many facets, but also because of
the tug-of-war between application-oriented instruction versus fundamental
concepts. Our work is an attempt to respond to the challenges of digital
media curriculum development by dividing the material into three modules:
a primer of basic material important to students regardless of the perspective
from which they approach digital media; an art module presenting concepts
and techniques important to art students; and a computer science module explaining
the mathematical, algorithmic, and technical underpinnings of digital media.
Each module consists of traditional text-based chapters accompanied by interactive
on-line demos, worksheets, and programming exercises as appropriate to the
subject. Parallel chapters run through all three modules in the areas of
digital imaging, digital audio, digital video, and multimedia programming.
Students who take digital media courses are interested in learning how to do photographic processing, audio/video production, and multimedia programming with the tools and languages of the day. For practical reasons, digital media curriculum needs to include specific application programs -- for example, Photoshop, Premiere, and Flash. However, focusing too narrowly on specific application programs has its drawbacks. One program works differently from another, even within the same medium (e.g. digital imaging or digital audio). In addition, versions of these programs change quickly, sometimes within a matter of months. Our goal is to create material that explains fundamental concepts of lasting utility, and to explain these concepts in a way that facilitates the students' ability to work in application programs and to be adaptable to new applications when they encounter them. For the primer and art module -- which is aimed at students who may have a non-technical or non-mathematical background -- this entails using simple analogies from everyday life, and showing how the concepts are important to their use of digital media tools in a generic explanation. For the computer science module -- which explains the mathematics and algorithms behind the scenes of digital media -- this entails identifying the material that is important to the students' advanced use of their tools. In the case of all three modules, it is important to identify the concepts that students most often misunderstand, and to find better ways to explain these.
An additional component of our digital media curriculum development project is an assessment of the pedagogical value of the material. Meaningful assessment is never easy in educational research, and in some ways we have found this part of our work to be the most difficult. The digital media course we teach is offered only once a year and typically has between 10 and 30 students in it -- a small population on which to base a significant statistical analysis. The small size of our classes also precludes doing comparative studies where one group uses our curriculum material while another uses other sources. In any case, such comparative studies entail treating groups of students unequally and risk disadvantaging one group in favor of another. Comparing student performance from one year to the next is another approach. But this method of assessment is slow-going, and results are difficult to interpret when comparison groups have different backgrounds and skill levels from the outset.
In developing an assessment plan, we are dealing with these problems in two ways. First, we are working with another university with large classes where comparative studies or pre/post testing might yield statistically significant results. Secondly, we are trying to devise a common-sense, practical approach to assessing our own students' performance based on observing what they do and do not understand. This paper describes an experiment recently conducted to help us develop our "in-house" assessment strategy.
2. The Assessment Experiment
Digital media courses involve
an integration of concepts and applications, and thus a proper assessment of
digital media curriculum material involves evaluating the students' understanding
of concepts and their application of concepts in hands-on activities. Thus,
one way to assess student performance might to look at the students' finished
work in projects assigned in a digital media course -- the images, audio files,
videos, or multimedia programs they create. However, it is not easy to determine
how well students understand concepts simply by looking at their completed
work. Digital media application programs generally have default settings and
powerful high-level commands that students can use -- through trial and error
-- to produce polished finished products even when they don't understand what
the settings mean. Also, there are multiple ways to get the same results, and
it is not possible to assess exactly what steps a student went through to accomplish
any given result. On the other hand, traditional tests and quizzes might assess
a student's understanding of concepts, but they reveal nothing about how the
student succeeds with hands-on activities. What is needed is an assessment
strategy that evaluates the students' understanding and application of concepts
in the context of using digital media tools.
This paper reports on our attempt to develop an appropriate strategy for assessing the pedagogical value of our curriculum material by experimenting with an assessment approach applied in the context of a learning unit on digital audio processing. The goals of this study follow:
The assignment upon which this paper is based was given to a digital media class of ten students in the spring 2004 semester. The students' learning objectives for the homework and lab exercise are described below:
The assignment had eight components -- seven items to be covered by students as homework, and the eighth constituting a closed lab worked on by three groups of three or four students per group. The homework assignments included the following:
The demos on Sound Fundamentals and Audio Dithering were on-line tutorials written in Macromedia Director/Shockwave. The worksheet on Non-Linear Companding was to be done in MatLab.
Text-Based Material:
(pdf,
~250 KB) Primer Chapter 4: Fundamentals of Digital Audio
(pdf,
~200 KB) Computer Science Chapter 4: Digital Audio
Pencil-and-Paper Worksheets:
(pdf,
~27 KB) Digital Audio Sampling Rate, Quantization, and File Size (computer
science module)
(pdf,
~15 KB) Conversion Between Air Pressure Amplitude and dB_SPL (computer science
module)
Multimedia Learning Aids
and Computer-Based Worksheets:
(Shockwave,
~ 460 KB) Sound Fundamentals (primer)
(Shockwave,
~ 600KB) Audio Dithering (computer science module)
(pdf,
~76 KB) Non-Linear Companding (computer science module)
The students were given a week and a half to complete the reading and worksheets. They then met for a two-hour closed lab session where they worked in homogeneous groups of three or four students, divided according to their test averages in an attempt to equalize the level of individual participation in each group. In the closed lab, the students completed an exercise using Adobe Audition, an application program with which the students had no previous experience. The exercise required that the students
3. Assessment of Student Learning
and Student Reaction to Curriculum Material
Students were
asked to rate the various components of the curriculum material
according to which one helped most in their understanding of each
concept. Students were also asked to indicate if they did not complete
one of the components. A component that was not completed by a
student received no rating from that student and did not affect
the average rating for that component. The ranking of the usefulness
of each component is given in the table below. Lower numbers indicate
a better rating; i.e., 1 means that the curriculum component was
considered most helpful by students, as determined by the average
rating. (Averages are not shown in the table.) The table also indicates
how many students did not complete each component.
The component most often not completed by the students was the MatLab exercise in Non-Linear Companding. In general, the students found the class lectures most helpful, with the text-based material a close second. With regard to their preference for the text-based material over the on-line material, we find it difficult to draw any conclusions. The on-line demos for the audio unit (Sound Fundamentals and Audio Dithering) are not the best ones among our curriculum material in the sense that they are fairly short and not as interactive as they could be. The exercise on Non-Linear Companding -- also an interactive computer-based exercise -- was carefully constructed and we expected it to be helpful in clarifying the concepts of µ-law encoding both graphically and mathematically. However, half the class failed to complete this exercise, probably because it was the last assignment handed out prior to the closed-lab, and also because the students were required to use MatLab to do the exercise, a program that some of them had not used before. When pressed for time, students omitted doing this exercise since they were not required to turn in their answer sheet for it.
Topic | Student Rating | Number of students who did NOT complete this component |
Sampling and Aliasing | ||
Primer Chapter 4: Fundamentals of Digital Audio | 2 | 0 |
CS Chapter 4: Digital Audio | 3 | 1 |
Interactive demo: Sound Fundamentals | 4 | 1 |
Class lecture | 1 | 0 |
Filters and Transforms | ||
CS Chapter 4: Digital Audio | 2 | 1 |
Class Lecture | 1 | 1 |
Quantizing, Quantization Error, Dithering | ||
Primer Chapter 4: Fundamentals of Digital Audio | 1 | 0 |
CS Chapter 4: Digital Audio | 3 | 1 |
Interactive demo: Sound Fundamentals | 4 | 1 |
Interactive demo: Audio Dithering | 5 | 1 |
Computer-Based Worksheet: Non-Linear Companding (MatLab) | 6 | 5 |
Class lecture | 2 | 0 |
Dynamic Range, Decibels | ||
Primer Chapter 4: Fundamentals of Digital Audio | 3 | 0 |
CS Chapter 4: Digital Audio | 2 | 1 |
Interactive demo: Sound Fundamentals | 4 | 1 |
Conversion between Air Pressure Amplitude and dB_SPL | 5 | 3 |
Class Lecture | 1 | 0 |
File Size, File Type, and Compression | ||
Primer Chapter 4:
Fundamentals of Digital Audio |
3 | 0 |
CS Chapter 4: Digital Audio | 2 | 1 |
Interactive demo: Sound Fundamentals | 4 | 1 |
Digital Audio Sampling
Rate, Quantization, and File Size |
5 | 1 |
Computer-Based Worksheet: Non-Linear Companding (MatLab) | 6 | 5 |
Class lecture | 1 | 0 |
Table 1. Students' rating of how much they learned from each learning unit.
In order to keep a record of the students' work and interactions in the closed lab, we recorded their screen activity with Camtasia and set up video cameras to record their group interactions as well. These videos helped us to pinpoint where students misunderstood concepts or had trouble navigating in the application program. For another example of video-analysis, see [Maor and Knibb, 1999].) Table 2 summarizes our analysis of the students' knowledge and reasoning as evidenced in the videos and their scores on the written answer sheets for the closed lab.
Topic | Corresponding Questions in Closed Lab Exercise | Group | Knowledge (based on group discussion) | Reasoning (based on group discussion) | Score per section (based on written answers) |
Sampling
and Aliasing |
questions 2,3 | group 1 | 4 | 4 | 6/6 |
group 2 | 4 | 4 | 6/6 | ||
group 3 | 4 | 3 | 5/6 | ||
Filters and Transforms | questions 1, 4-6 | group 1 | 3 | 4 | 9/9 |
group 2 | 2 | 3 | 7.5/9 | ||
group 3 | 2 | 1 | 5.5/9 | ||
Quantizing, Quantization Error, Dithering | questions 7-11 | group 1 | 4 | 4 | 10/10 |
group 2 | 3 | 3 | 10/10 | ||
group 3 | 3 | 2 | 10/10 | ||
Dynamic Range, Decibels | questions 12-15 | group 1 | 4 | 3 | 6/6 |
group 2 | 4 | 3 | 6/6 | ||
group 3 | 2 | 2 | 5/6 | ||
File Size, File Type, Compression | questions 16-19 | group 1 | 4 | 4 | 4/4 |
group 2 | 4 | 4 | 4/4 | ||
group 3 | 3 | 3 | 4/4 |
Table 2. Assessment of students' knowledge and reasoning as evidenced by the closed lab worksheet.
In many cases, the closed lab exercise appeared to be successful not only in demonstrating the students' mastery of the concepts learned in the homework, but also in reinforcing these concepts by requiring the students to reason through the questions. The Camtasia and video excerpts below show cases where the lab exercise was successful in directing the students toward sound reasoning and better understanding of the concepts.
The concepts or features of the audio editing program that students had trouble with are listed below. Some are illustrated with Camtasia or video excerpts. (Note that some activities listed above as "successful" are also listed below as causing trouble for students. This is because the groups did not do equally well on all activities and questions).
From these observations, we determined that we needed to revised our curriculum material to include a more detailed explanation of windowing functions for the fast Fourier transform; additional reinforcement of the difference between sampling and quantization in the context of digital audio; a clearer explanation of the implications of dynamic range; and more reinforcement of the interpretation of spectral views. We would like to illustrate these concepts in short interactive tutorials with questions that engage the students' active learning.
The Camtasia and video
excerpts:
Video
clip (QuickTime, ~17 MB): understanding the effects of aliasing.
Screen
capture (Flash, ~1 MB): successful mastery of applying a notch filter to
filter out an aliased sound wave component resulting from downsampling.
Video
clip (QuickTime, ~8.5 MB): correctly interpreting a spectral view.
Screen
capture (Flash, ~1.5 MB): getting a better understanding of dynamic range.
Video
clip (QuickTime, ~10 MB): understanding and hearing benefits of µ-law
encoding in reducing file size while preserving dynamic range.
Video
clip (QuickTime, ~13 MB): understanding how to read the spectral view of
a wave form.
Video
clip (QuickTime, ~27 MB): confusing the effects of sampling rate reduction
(aliasing) with the effects of bit depth reduction (quantization error and
distortion), and misunderstanding the implications of dynamic range as restricted
by bit depth.
Video
clip (QuickTime, ~5 MB):understanding how dithering works -- i.e., does it "smooth" a
sound wave or make it more irregular.
4. Conclusions Related to Future
Plans for Assessment
Our conclusions
from the experiment are the following:
5. References
Yue-Ling Wong, Jennifer
Burg, and Victoria Strokanova, "Digital Media in the Computer Science
Curricula," Proceedings of 35th SIGCSE: Technical Symposium
on Computer Science Education, Norfolk, Virginia, March 3-7, 2004, pp.
427-430.
Maor, Dorit, and Ken Knibb, "Video Analysis: A Qualitative Tool for Investigating Students' Learning in a Constructivist-Oriented Multimedia in a Science Classroom, http://www.aare.edu.au/99pap/mao99401.htm.
6. Acknowledgments
This material is based upon
work supported by the National Science Foundation under Grant No. DUE-0127280
and DUE-0340969.
********** End of Document **********
© 2004 Wake Forest University (from Volume 6, Number 1, of The Interactive Multimedia Electronic Journal of Computer-Enhanced Learning).