Effective Use of Audio Media in Multimedia Presentations

Presented by Brenda Kerr

Middle Tennessee State University

Abstract
Why Should Educators Include Audio Media in their Presentations?
Learning Styles
Examples: Effective Use of Audio in Multimedia Presentations
Literal Sounds and Non-Literal Sounds
Functions of Sound as Used in Radio and Television
Roles of Audio Media in Multimedia Production
Aesthetic Factors of Sound
Software
Audio File Types and Platform Compatibility
References

This paper emphasizes research-based reasons for adding audio to multimedia presentations. Media examples, links for gaining more in-depth knowledge concerning this topic and procedures for adding audio media from any PC (Macintosh or Windows-based) can be referenced from the following URL: http://www.mtsu.edu/~itres.

This topic was chosen because the author has observed many faculty members adding still images, video, and links to web and non-web documents to their presentations but few faculty members have taken advantage of the learning provided by audio media integration. In preparation for designing a class around this topic the author has begun the following investigation into research in the effective use of audio media in multimedia presentations.

[Top]

Why Should Educators Include Audio Media in their Presentations?

Thompson, Simonson, and Hargrave in Educational Technology, A Review of the Research (AECT) reviewed studies conducted on the effectiveness of audio media on learning when accompanied by other forms of media. Their review of the studies offered the following suggestions :

• Students can learn when various forms of audio media accompany other media.
• The use of background music can increase achievement for some learners, but is probably not necessary.
• The use of audio media with other media may enhance the understanding of content material.
• The meaning of a visual message is often ambiguous and subject to personal interpretation. The use of words to direct attention is essential.
• With visuals, some verbalization is better than no visuals, but there is no optimum amount. Slow speeds for transmitting verbal information are favored but they can be too slow. Rates need to be tailored to fit the student and their familiarity with the content.
• When narration is accompanied by video the optimum rate of the narration appears to be slower.
• The audio channel is much more capable of maintaining attention if it is used as an interjection on the visual channel rather than being continuously parallel with the visual.

Types of Research Studies

Evaluation Research is usually the first type of research done for each media type. Evaluation Research tries to determine whether people can learn from a particular form of media. It was found that given favorable conditions, students could learn from any instructional media. Media comparison studies were then conducted to try to determine if one media type was more effective in learning than another type (1920’s – 1960’s). Media comparison studies produced insignificant results. For every study that showed that a new medium was better, another study showed the opposite .

Intra-medium studies were the next type of research conducted. Intra-medium studies examined the interactions among student, task, and specific media characteristics in terms of what happens when these variables were manipulated. They compare alternative methods of using a particular medium. According to Thompson, Simonson, and Hargrave the design of these studies were based on Saloman’s

observation that the effectiveness of a medium depends on the nature of the instruction. The major research question in these studies was "Which are the most effective instructional approaches using this medium?" A particular medium was used in all groups participating in the study. The independent variable was the instructional approach, not the medium itself .

The next type of research conducted was Aptitude Treatment Interaction Studies. Aptitude Treatment Interaction Studies attempted to take into account student aptitudes in the research design. Media researchers accepted a new paradigm. This paradigm acknowledged the interaction that occurs between external stimuli (presented by media) and internal cognitive processes that support learning (Clark, 1988). Information about a learner was helpful in adapting instruction in order to provide an environment in which particular learners can thrive. Clark tells us that media by themselves do not affect learning but rather it is the particular qualities of media or a specific medium that "affects particular cognitive processes that are relevant for students with specific aptitudes to learn particular knowledge or skills". Research in Aptitude Treatment Interaction Studies have led people to recognize "the importance of different learning styles and methods of processing information as well as the correlation that exists between learner variables and content treatments.

[Top]

Learning Styles

Gordon (1998) researched the relevance of learner characteristics and learning styles when planning law school courses. He identified the four major learning modalities, kinesthetic, tactual, auditory, and visual. Kinesthetic learners and auditory learners seemed to benefit the most by the integration of audio media into teaching strategies. Gordon’s table below provides a list of suggested teaching techniques based on learning modalities. He does not provide data on the tactual learner .

Modality	Teaching technique
Kinesthetic	This student needs a combination of stimuli. The manipulation of material along with the accompanying sights and sounds (words and numbers seen and spoken) will make a big difference to this student.
Auditory	This student will benefit from hearing audio tapes, rote oral practice, lecture or a class discussion. This student will benefit from tutoring another or delivering an explanation to his/her study group or to the teacher.
Visual	This student will benefit from worksheets, workbooks, and texts. Given some time alone with a book, this student may learn more than the class.

Examples: Effective Use of Audio in Multimedia Presentations

Lee Daniels (1995), in his paper Audio Vision: Audio-Visual Interaction in Desktop Multimedia lists the three elements of sound used in multimedia presentations. He explains how these elements should be used to stimulate learning through the functions they serve and roles they provide. Daniels’ ideas are summarized in the next few pages.

Elements of sound

The three audio elements in multimedia production are speech, sound effects, and music. Silence and its effects also need to be considered in planning projects. The various functions of these elements are discussed in the next three sections of this paper. It is important that the educator use care in considering the audience when using audio media to evoke source images in students’ minds. If the students have not been thoroughly exposed to the source that produces the sound the effect and learning will be lost.

Audio Element: Speech

Narration, dialogue, and direct address are three functions of the speech element. Their functions will be discussed below.

Narration

Narrative speech can be used to:

• Deliver concrete information: Concrete information could include directions for completing a project or possibly descriptive information that relates to the image being displayed. When narration is presented with text, the text and narration must be exactly the same. Discrepancies may result in distraction and cause interference in learning the material which translates to less retention or misinterpretation of material.
• Replace text:
  Narration is most useful, as a replacement of text when screen space is limited and the addition of text would reduce the visual impact of the page. Narration saves screen space and visual clutter.
• Direct viewer’s attention:
  The image displayed on the screen may need to be the focal point. Narration is used to direct viewer attention to the image being displayed rather than forcing the viewer to alternate between viewing an image and reading the text explanation.

Narration and dialogue together can:

Affect intensity by setting the pace. Pace effects the intensity of emotion during a presentation.

Examples include:

• Fast moving narration adds to the intensity of time lapsed animation. Slow moving narration complements the sober mood of a funeral.
• Fast paced dialogue between two characters can reflect tension, anger, excitement, or nervousness. Smooth, even paced dialogue reflects friendliness, relaxation, and confidence.

Affect the listener’s perception through changes in tone quality.

Examples include:

• Bright and present narration is perceived to be closer and more intimate and trustworthy.
• Speech that sounds dull and distant would have the opposite effect.

Direct Address

Direct address refers to the character speaking directly to the audience.

Examples include:

• Some TV commercials that talk directly to the audience.
• Speeches
• Newscasts

Audio Element: Sound Effects

Sound effects can function contextually and narratively.

Contextual Function

When sound effects have a contextual function the sound effect interprets the visual as it appears.

Examples include:

• a dog barking or a dog begging for a treat
• the roar of a jet or airplane engine in normal flight or taking off, or the sound of a jet having engine trouble

Narrative Function

When sound effects have a narrative function the sound effect adds more to the image’s apparent information. The functions of the narrative format can be broken down even further into those that provide descriptive effects and those that serve a commentary function.

Narrative Function: Descriptive Effect

Sound effects contribute to the subtle aspects of an image. Subtle aspects are those features that are hard to define or perceive but that contribute greatly to the emotional effect of the image or scene.

Examples include:

• The sounds of gentle ocean surf which may include gulls, people playing, and boat sounds used to set a particular mood.
• The sound of a violent ocean surf and warning sirens sounding in the background. The mood of this example would be quite different than the mood of the first example.
• Imagine a picture of a metal triangle being struck with a metal beater. The audience may hear a clink indicating that the triangle may be made of inferior materials or they may hear a clear ringing sound indicating that the triangle is made out of quality materials. Perhaps the audience only hears a thud, indicating that the triangle only looked like it was made of metal but might have actually been made of wood or plastic.
• Imagine a picture of a cymbal player crashing a pair of cymbals. If a clear ringing sound is heard the teacher knows that the player is holding the cymbals correctly as they are being played. If a muffled crash is heard followed by little ringing the teacher assumes that the player is not holding the cymbals correctly and may be placing too much of his/her hands on the cymbals as they are being played.

Narrative Function: Commentative Sounds

Commentative sounds also tell more about an image but the information is usually unrelated to the visual itself.

Example:

• "Imagine a program about air pollution and a scene of city traffic. Treating and blending the car engines to ‘sputter’ and ‘cough’ comments on the detrimental effects that air pollution has on the air we breathe" .

Audio Element: Music

Music is very effective in communicating complicated emotions and moods. Functions of music in multimedia presentations include establishing locale or time, identifying characters and events, acting as a transition element between contrasting scenes, and setting the mood and pace of presentations. Examples of each function are listed below .

• Locale: Music can define a locale with ethnic melodies.
• Time: Music can establish time with musical elements that suggest a period in history such as the 1960’s or the Roman era.
• Identification: Music can identify characters and events with recurring themes. A short musical phrase or specific sound effect can be used to signal the appearance of a person, action, or situation. This is sometimes called leitmotiv (German for "leading motive") .
• Transitions: Music can be used to connect one idea or scene to another. It can also smoothe the transition to a contrasting theme. It prepares the audience by letting them know that something is going to change.
• Pace: Music can be used to establish the pace of the presentation. This pace can parallel the visual media or provide counterpoint to signify tension or irony.

Silence

Every moment of a multimedia presentation does not need to be filled with sound. Silence can be used to set a mood or to provide a moment for reflection. Producers of multimedia presentations need to consider the use of silence as well as the use of the other audio elements when designing a presentation. As stated earlier in this paper, the audio channel is much more capable of maintaining attention if it is used as an interjection on the visual channel rather than being continuously parallel with the visual.

 [Top]

Literal Sounds and Non-Literal Sounds

Zettl (1999), in his text Sight Sound Motion, Applied Media Aesthetics breaks down sounds into two categories, literal sounds and non-literal sounds. Literal sounds include speech and environmental sounds. Non-literal sounds include background music and other sounds that seem to influence feeling in some way . All of the sound elements listed in the previous section can be used in a literal or non-literal way. Think about how they can be used as you read the definitions below.

Literal Sounds

Literal sounds are referential. They convey a specific literal meaning. They refer the listener to the sound-producing source. Conversations or the sound of rush-hour traffic refer to people and automobiles. Literal sounds can be source-connected or source-disconnected. The audience can see the sound-producing source when they hear source-connected sounds. The source of source-disconnected sound is off-screen. The listener visualizes the source of the source-disconnected sound.

Non-Literal Sounds

Non-literal sounds are not intended to refer to a particular source or convey literal meaning. They are deliberately source-disconnected and do not evoke a visual image of the sound-producing source. The following sound effects, boings, hisses, and whams, usually used in cartoons, are non-literal sounds. Romantic music played in the background during a love scene or the rhythmic musical themes behind newscasts are also non-literal sounds. Non-literal sounds are called "nodiegetic" which means they occupy non-story space .

Summary

Non-literal and literal sounds are often combined in the same scene. Sounds can be literal or non-literal depending on their context within the presentation or scene. Music played in the background to influence the mood of a scene would be considered non-literal sound. Music played, as the audience watches the orchestra musicians perform, would be considered literal sound.

[Top]

Functions of Sound as Used in Radio and Television

Zettl, (1999) has broken down the use of sound in radio and television production into three categories. He tells us that sound functions to provide information, establish outer orientation, and establish inner orientation . These functions should also be considered in multimedia production.

Provide information

Zettl established that information is imparted through speech in the form of dialogue, direct address, and narration. Direct address refers to the character speaking directly to the audience. We see examples of direct address everyday on TV in commercials, speeches, and newscasts. Earlier in this paper Daniels’ functions of the audio element, speech, are listed as narration and dialog. Note that Zettl has added the function of direct address. The function narration can be source-connected (on camera) or source-disconnected (off camera).

Establish Outer Orientation

Sound establishes the outer orientation of a scene by orientating the scene in space, time, situation, and external event condition. Orientation in space refers to the location, the spatial environment (sounds produced in a large room sound different than those produced in a small room) and off-screen space (sounds of people and objects not seen on-screen). Morning sounds such as an alarm clock ringing and bacon frying distinguishes the time of day. Situation orientated sounds refer to those that describe a specific situation. These sounds may be predictive such as those used to indicate the recurring appearance of a person or action (leitmotiv), or they may signify the coming of an event such as danger (non-recurring).

Establish Inner Orientation

Sounds that establish inner orientation set the mood, establish the internal condition, and provide energy and structure to the scene. Mood can be set using music or some sort of non-musical, electronically produced sound. Sounds that indicate an unstable environment or the feelings of a person establish the internal condition. Sounds provide or increase the aesthetic energy of a scene. This energy affects the emotions of the viewers.

Structure is the interaction between the rhythmic structure of the sound versus that of the visual source. The interplay between the video and audio structures can be parallel, irregular, or highly independent of each other. If the sound and visual structure are considered parallel they move at the same rate. In irregular structures the rates of the visual and audio structures may vary. They may move together for a period of time and then move separately. If the visual and audio elements move independently of one another they provide contrast and interaction among the media types. \

[Top]

Roles of Audio Media in Multimedia Production

Daniels lists four roles that audio media play in multimedia production. These roles are: picture defines sound, sound defines picture, sound parallels picture, and sound counterpoints picture. Each role is described below. Examples are provided .

Picture defines sound

When a picture defines a sound the sound is defined when the visual image is so strong that the accompanying sound is a literal translation of the image. This is very similar to the descriptive function of sound effects described earlier in this paper.

• Example:

• A raging storm, with crashing waves and bent palms demands a soundtrack that consists of wind, surf, and rain sound effects. Audio is supportive of the dominant visual, reinforcing the image.
• A quiet beach with small calm waves demands a soundtrack that consists of soft surf sounds and sounds of children playing.

Sound defines picture

When the sound defines the picture the sound is so distinctive that the listener forms an image of the source in his or her mind before the image is displayed. Examples are listed below. Remember to consider the background knowledge of your students when you use sound in this manner.

• Multimedia program on the Brazilian rain forest

• A still image of the jungle interior is accompanied by the solitary sounds of the environment:

• Rainfall
• Bird calls
• Other animals
• Lively ethnic music
  

• Distinctive Sounds Produced by Objects

• Sound of chain saws
• Sound of various types of machinery

Sound parallels picture

Sound parallels picture is the most common relationship between audio and visual elements. The audio element combines with the visual element to create a mood or deliver information that is more potent than either element alone .

• Example: The sounds of battle with gunshots, cannon, and anguished screams complement the visual of a battle scene. The ferocity and destruction of war is conveyed by both media separately, but is intensified by both elements together.

Sound counterpoints picture

Sound counterpoints the visual image when both media elements contain unrelated information that creates an effect that is not conveyed by either media element alone. Previously Zettl (1999) told us that visual and audio elements moving independently of each other provide contrast and interactivity among media types.

• Example: In a presentation on the civil rights movement, irony is created when a visual montage of segregated public facilities is underscored by a reading of the United States Constitution.

[Top]

Aesthetic Factors of Sound

There are three basic aesthetic factors of sound use in any scene. They are figure ground, sound perspective, and sound continuity. Figure ground is the most important sound in the scene, the sound that is emphasized. All other sounds will be in the background. Sound perspective refers to close-up sounds matching close up visuals and distant sounds matching distant visuals. Sound continuity is established when the intended volume and quality of sound is maintained over a series of events. Sound continuity is usually achieved by maintaining the same level of ambient (background) sounds. The figure ground may vary in intensity but the background sounds provide the continuity.

[Top]

Software

Software for Automating Online Multimedia Presentations

• Synchronized Multimedia Integration Language (SMIL): http://webreview.com/wr/pub/1999/03/12/feature/index.html
• RealAudio/Video: http://www.realaudio.com
• Top Class: http://www.wbtsystems.com/
• The Sync-O-Matic 3000: http://www.egr.msu.edu/~crs/projects/syncomat/

[Top]

Audio File Types and Platform Compatibility

• Editing Visual and Audio Media Class (MTSU): http://www.mtsu.edu/~itres/itt/VisualAudioHandout/page7.html
• Duquesne University’s Digital Duke: http://the-duke.duq-duke.duq.edu/notes/LECTURES/sec2.htm

References:

Daniels, L. (1995). Audio-Vision: Audio-Visual Interaction in Desktop Multimedia. Annual Conference of the International Visual Literacy Association, Tempe, Arizona.

Gordon, H. R. D. (1998). Selected Instructional Delivery Methods and Teaching Techniques for Teaching School Law Courses. American Educational Research Association, San Diego, CA.

Thompson, A. D., Simonson, Michael R., Hargrave, Constance P. (1992). Educational Technology: A Review of the Research. Washington D.C.: Association for Educational Communications and Technology.

Zettl, H. (1999). Sight Sound Motion, Applied Media Aesthetics. San Francisco: Wadsworth Publishing Company.

[Top]