Printer-friendly version

Developing a Strategy for Creating and Assessing Digital Media Curriculum Material
Jennifer Burg, Wake Forest University
Yue-Ling Wong, Wake Forest University
Leah McCoy, Wake Forest University

Abstract
Creating curriculum material in digital media is a challenge, not only because of its many facets, but also because of the tug-of-war between application-oriented instruction versus fundamental concepts. Our work is an attempt to respond to the challenges of digital media curriculum development by dividing the material into three modules with parallel chapters: a primer, an art module, and a computer science module. An additional component of our digital media curriculum development project is an assessment of the pedagogical value of the instructional modules we have created. One of the special challenges in assessing digital media curriculum material is making the connection between concepts and applications. The two traditional methods of assessing student performance -- evaluating either their final projects or their mastery of concepts through tests and quizzes -- fail to associate "knowing" with "doing." This paper describes an assessment experiment recently conducted to help us gain insight into students' thought processes. Students are observed and videotaped in a closed lab designed to help them understand concepts and to help us assess how they relate concepts to activities. Based on our observations, we propose a refinement to our assessment procedures and improvements to our curriculum material.

About the authors...

1. Background 
Digital media is an inherently interdisciplinary area of study with links to computer science, art, film, music, communications, architecture, and design [Burg, Wong, and Strokanova 2004]. Creating appropriately targeted curriculum material in digital media is a challenge, not only because of its many facets, but also because of the tug-of-war between application-oriented instruction versus fundamental concepts. Our work is an attempt to respond to the challenges of digital media curriculum development by dividing the material into three modules: a primer of basic material important to students regardless of the perspective from which they approach digital media; an art module presenting concepts and techniques important to art students; and a computer science module explaining the mathematical, algorithmic, and technical underpinnings of digital media. Each module consists of traditional text-based chapters accompanied by interactive on-line demos, worksheets, and programming exercises as appropriate to the subject. Parallel chapters run through all three modules in the areas of digital imaging, digital audio, digital video, and multimedia programming.

Students who take digital media courses are interested in learning how to do photographic processing, audio/video production, and multimedia programming with the tools and languages of the day. For practical reasons, digital media curriculum needs to include specific application programs -- for example, Photoshop, Premiere, and Flash. However, focusing too narrowly on specific application programs has its drawbacks. One program works differently from another, even within the same medium (e.g. digital imaging or digital audio). In addition, versions of these programs change quickly, sometimes within a matter of months. Our goal is to create material that explains fundamental concepts of lasting utility, and to explain these concepts in a way that facilitates the students' ability to work in application programs and to be adaptable to new applications when they encounter them. For the primer and art module -- which is aimed at students who may have a non-technical or non-mathematical background -- this entails using simple analogies from everyday life, and showing how the concepts are important to their use of digital media tools in a generic explanation. For the computer science module -- which explains the mathematics and algorithms behind the scenes of digital media -- this entails identifying the material that is important to the students' advanced use of their tools. In the case of all three modules, it is important to identify the concepts that students most often misunderstand, and to find better ways to explain these.

An additional component of our digital media curriculum development project is an assessment of the pedagogical value of the material. Meaningful assessment is never easy in educational research, and in some ways we have found this part of our work to be the most difficult. The digital media course we teach is offered only once a year and typically has between 10 and 30 students in it -- a small population on which to base a significant statistical analysis. The small size of our classes also precludes doing comparative studies where one group uses our curriculum material while another uses other sources. In any case, such comparative studies entail treating groups of students unequally and risk disadvantaging one group in favor of another. Comparing student performance from one year to the next is another approach. But this method of assessment is slow-going, and results are difficult to interpret when comparison groups have different backgrounds and skill levels from the outset.

In developing an assessment plan, we are dealing with these problems in two ways. First, we are working with another university with large classes where comparative studies or pre/post testing might yield statistically significant results. Secondly, we are trying to devise a common-sense, practical approach to assessing our own students' performance based on observing what they do and do not understand. This paper describes an experiment recently conducted to help us develop our "in-house" assessment strategy.

2. The Assessment Experiment
Digital media courses involve an integration of concepts and applications, and thus a proper assessment of digital media curriculum material involves evaluating the students' understanding of concepts and their application of concepts in hands-on activities. Thus, one way to assess student performance might to look at the students' finished work in projects assigned in a digital media course -- the images, audio files, videos, or multimedia programs they create. However, it is not easy to determine how well students understand concepts simply by looking at their completed work. Digital media application programs generally have default settings and powerful high-level commands that students can use -- through trial and error -- to produce polished finished products even when they don't understand what the settings mean. Also, there are multiple ways to get the same results, and it is not possible to assess exactly what steps a student went through to accomplish any given result. On the other hand, traditional tests and quizzes might assess a student's understanding of concepts, but they reveal nothing about how the student succeeds with hands-on activities. What is needed is an assessment strategy that evaluates the students' understanding and application of concepts in the context of using digital media tools.

This paper reports on our attempt to develop an appropriate strategy for assessing the pedagogical value of our curriculum material by experimenting with an assessment approach applied in the context of a learning unit on digital audio processing. The goals of this study follow:

1. To evaluate how well the students could navigate through an unfamiliar application program, understanding the concepts they had learned as these concepts were applied in a hands-on setting.
2. To note the concepts that still gave students trouble.
3. To identify concepts that we had not explained in the curriculum material but that were important to their effective use of the application tools.
4. To evaluate which of the curriculum approaches (text-based, worksheets, on-line demos, in-class lecture) the students found most useful to their understanding.
5. To apply what we learned in our experiment to developing a general strategy for continued development and assessment of digital media curriculum material.

The assignment upon which this paper is based was given to a digital media class of ten students in the spring 2004 semester. The students' learning objectives for the homework and lab exercise are described below:

1. To learn basic concepts of digital audio, including sampling rate, aliasing, quantization, quantization error, dithering, frequency analysis, dynamic range, audio compression, and common audio file types.
2. To gain practical experience with the concepts listed above in the context of a digital audio editing program.

The assignment had eight components -- seven items to be covered by students as homework, and the eighth constituting a closed lab worked on by three groups of three or four students per group. The homework assignments included the following:

• Text-Based Material
  • Primer Chapter 4: Fundamentals of Digital Audio
  • Computer Science Chapter 4: Digital Audio
• Pencil-and-Paper Worksheets
  • Digital Audio Sampling Rate, Quantization, and File Size (computer science module)
  • Conversion between Air Pressure Amplitude and dB_SPL (computer science module)
• Multimedia Learning Aids and Computer-Based Worksheets
  • Sound Fundamentals (primer)
  • Audio Dithering (computer science module)
  • Non-Linear Companding (computer science module)

The demos on Sound Fundamentals and Audio Dithering were on-line tutorials written in Macromedia Director/Shockwave. The worksheet on Non-Linear Companding was to be done in MatLab.

Text-Based Material:
(pdf, ~250 KB) Primer Chapter 4: Fundamentals of Digital Audio
(pdf, ~200 KB) Computer Science Chapter 4: Digital Audio

Pencil-and-Paper Worksheets:
(pdf, ~27 KB) Digital Audio Sampling Rate, Quantization, and File Size (computer science module)
(pdf, ~15 KB) Conversion Between Air Pressure Amplitude and dB_SPL (computer science module)

Multimedia Learning Aids and Computer-Based Worksheets:
(Shockwave, ~ 460 KB) Sound Fundamentals (primer)
(Shockwave, ~ 600KB) Audio Dithering (computer science module)
(pdf, ~76 KB) Non-Linear Companding (computer science module)

The students were given a week and a half to complete the reading and worksheets. They then met for a two-hour closed lab session where they worked in homogeneous groups of three or four students, divided according to their test averages in an attempt to equalize the level of individual participation in each group. In the closed lab, the students completed an exercise using Adobe Audition, an application program with which the students had no previous experience. The exercise required that the students

• create pure tones, listen to them, and view and interpret the frequency analysis graph
• add pure tones into one composite waveform, and look at the waveform and frequency analysis graph
• downsample an audio file to produce aliasing, and identify the aliased wave in the frequency analysis graph
• filter out the aliased frequency
• consider how an audio file could be stretched without a change of frequency
• analyze the spectral view of a sound wave
• reduce the bit depth of an audio sample and observe and hear the effects of quantization error
• observe and analyze the effects of dynamic range as determined by bit depth
• observe and analyze the effect of µ-law encoding and audio compression on file size and sound quality

3. Assessment of Student Learning and Student Reaction to Curriculum Material
Students were asked to rate the various components of the curriculum material according to which one helped most in their understanding of each concept. Students were also asked to indicate if they did not complete one of the components. A component that was not completed by a student received no rating from that student and did not affect the average rating for that component. The ranking of the usefulness of each component is given in the table below. Lower numbers indicate a better rating; i.e., 1 means that the curriculum component was considered most helpful by students, as determined by the average rating. (Averages are not shown in the table.) The table also indicates how many students did not complete each component.

The component most often not completed by the students was the MatLab exercise in Non-Linear Companding. In general, the students found the class lectures most helpful, with the text-based material a close second. With regard to their preference for the text-based material over the on-line material, we find it difficult to draw any conclusions. The on-line demos for the audio unit (Sound Fundamentals and Audio Dithering) are not the best ones among our curriculum material in the sense that they are fairly short and not as interactive as they could be. The exercise on Non-Linear Companding -- also an interactive computer-based exercise -- was carefully constructed and we expected it to be helpful in clarifying the concepts of µ-law encoding both graphically and mathematically. However, half the class failed to complete this exercise, probably because it was the last assignment handed out prior to the closed-lab, and also because the students were required to use MatLab to do the exercise, a program that some of them had not used before. When pressed for time, students omitted doing this exercise since they were not required to turn in their answer sheet for it.

In order to keep a record of the students' work and interactions in the closed lab, we recorded their screen activity with Camtasia and set up video cameras to record their group interactions as well. These videos helped us to pinpoint where students misunderstood concepts or had trouble navigating in the application program. For another example of video-analysis, see [Maor and Knibb, 1999].) Table 2 summarizes our analysis of the students' knowledge and reasoning as evidenced in the videos and their scores on the written answer sheets for the closed lab.

In many cases, the closed lab exercise appeared to be successful not only in demonstrating the students' mastery of the concepts learned in the homework, but also in reinforcing these concepts by requiring the students to reason through the questions. The Camtasia and video excerpts below show cases where the lab exercise was successful in directing the students toward sound reasoning and better understanding of the concepts.

• understanding the effects of aliasing
• successful mastery of applying a notch filter to filter out an aliased sound wave component resulting from downsampling
• correctly interpreting a spectral view
• getting a better understanding of dynamic range as determined by bit depth by looking closely at samples and decibel ranges for a sound file with reduced bit depth
• understanding and hearing benefits of µ-law encoding in reducing file size while preserving dynamic range

The concepts or features of the audio editing program that students had trouble with are listed below. Some are illustrated with Camtasia or video excerpts. (Note that some activities listed above as "successful" are also listed below as causing trouble for students. This is because the groups did not do equally well on all activities and questions).

• how the fast Fourier transform works to create the frequency analysis view, and why a pure tone, expected to have only one frequency component, shows multiple other low amplitude frequency components (reasoning)
• what a windowing function is with regard to the fast Fourier transform (knowledge)
• how to filter out a frequency component (reasoning)
• understanding how to read the spectral view of a wave form (knowledge)
• seeing the effects of quantization error with or without dithering by reading and interpreting the spectral view of a waveform (reasoning)
• confusing the effects of sampling rate reduction (aliasing) with the effects of bit depth reduction (quantization error and distortion), and misunderstanding the implications of dynamic range as restricted by bit depth (thinking incorrectly that dynamic range determines how loud and soft the sounds can be) (knowledge and reasoning)
• understanding how dithering works -- i.e., does it "smooth" a sound wave or make it more irregular

From these observations, we determined that we needed to revised our curriculum material to include a more detailed explanation of windowing functions for the fast Fourier transform; additional reinforcement of the difference between sampling and quantization in the context of digital audio; a clearer explanation of the implications of dynamic range; and more reinforcement of the interpretation of spectral views. We would like to illustrate these concepts in short interactive tutorials with questions that engage the students' active learning.

The Camtasia and video excerpts:

Video clip (QuickTime, ~17 MB): understanding the effects of aliasing.

Screen capture (Flash, ~1 MB): successful mastery of applying a notch filter to filter out an aliased sound wave component resulting from downsampling.

Video clip (QuickTime, ~8.5 MB): correctly interpreting a spectral view.

Screen capture (Flash, ~1.5 MB): getting a better understanding of dynamic range.

Video clip (QuickTime, ~10 MB): understanding and hearing benefits of µ-law encoding in reducing file size while preserving dynamic range.

Video clip (QuickTime, ~13 MB): understanding how to read the spectral view of a wave form.

Video clip (QuickTime, ~27 MB): confusing the effects of sampling rate reduction (aliasing) with the effects of bit depth reduction (quantization error and distortion), and misunderstanding the implications of dynamic range as restricted by bit depth.

Video clip (QuickTime, ~5 MB):understanding how dithering works -- i.e., does it "smooth" a sound wave or make it more irregular.

4. Conclusions Related to Future Plans for Assessment
Our conclusions from the experiment are the following:

• We plan to continue with the common sense strategy of identifying the concepts that students find most troublesome and refining the curriculum material accordingly. We will continue the refinement process during the remainder of the grant period while the curriculum material is being developed.
• Not all the students completed all the curriculum material before the closed lab. Meaningful assessment of the effectiveness of curriculum material is not possible if we can't be sure that the students actually went through the material. We have been using checklists and the honor system to determine what material the students have actually covered, but we may need a tighter system for accountability. One possibility being considered is to have a worksheet or set of exercises associated with every curriculum component. (For example, every on-line demo would be coupled with a worksheet.) The worksheet would have to be turned in after the component was studied by the student. Students would be more likely to cover the material if they had to complete and submit an associated worksheet.
• The closed lab exercise used in this experiment was fairly structured and directive in nature in that students were given specific steps to go through and asked to analyze what they saw or heard. However, they didn't do much independent exploring of Adobe Audition, and they didn't actually produce an audio product of any significance. We are considering more free-form exercises which describe a digital media product to be created (an image, audio file, video, etc.) for a particular purpose in a particular context with given constraints. The students would then have to navigate around an application program and figure out how to accomplish their goal. By observing their activity, we would try to identify concepts that students needed to be clarified in the curriculum material to help them in their active use of application programs.
• The previous point raises the issue of associating a lab with the digital media course. While we use only very limited classroom instruction time focusing on particular application programs, it would be helpful to have a weekly closed lab in which to help students with these programs as they complete projects. This would also be a setting in which we could observe their hands-on activities and thereby get ideas for refining the curriculum material.

5. References
Yue-Ling Wong, Jennifer Burg, and Victoria Strokanova, "Digital Media in the Computer Science Curricula," Proceedings of 35^th SIGCSE: Technical Symposium on Computer Science Education, Norfolk, Virginia, March 3-7, 2004, pp. 427-430.

Maor, Dorit, and Ken Knibb, "Video Analysis: A Qualitative Tool for Investigating Students' Learning in a Constructivist-Oriented Multimedia in a Science Classroom, http://www.aare.edu.au/99pap/mao99401.htm.

6. Acknowledgments
This material is based upon work supported by the National Science Foundation under Grant No. DUE-0127280 and DUE-0340969.

********** End of Document **********

IMEJ multimedia team member assigned to this paper	Yue-Ling Wong

© 2004 Wake Forest University (from Volume 6, Number 1, of The Interactive Multimedia Electronic Journal of Computer-Enhanced Learning).