How to sonify data in your C or C++ application, using VSS.

Camille Goudeseune

1. Overview

This document explains how to perform "data sonification" in your application.
It assumes that your application is written in C or in C++, and that you have a copy of VSS available, version 3.1 dated 1/1/1999 or later.
It assumes that you've managed to get basic sound running as described here.

Sonifying data here means making sounds which:

2. Where will sound improve my application?

2.1 A checklist.

  1. Does the application have several different "modes" it can be in?
    Play different background sounds to remind the user which mode they're in.

  2. Do the GUI controls take effort to point at or adjust?
    Trigger short sounds to give feedback for selecting and releasing a control; maybe, play a continuous controllable sound while the control is being adjusted.

  3. Do emergencies arise, where the user of your application needs to be interrupted from their main task to attend to the abnormality?
    Have the emergency trigger an alarm sound in addition to any visual alert: the sound will capture the user's attention more strongly.

2.2 Should I use audio or video for a particular message?

Use auditory presentation if: Use visual presentation if:
The message is simple. The message is complex.
The message is short. The message is long.
The message will not be referred to later. The message will be referred to later.
The message deals with events in time. The message deals with events in space.
The message calls for immediate action. The message does not call for immediate action.

Derived from:
B.H. Deatherage, "Auditory and Other Sensory Forms of Information Presentation." In Van Cott and Kinkade (Eds.), Human Engineering Guide to Equipment Design, Revised Ed., Washington: U.S. Gov't Printing Office, 1972, p. 124.

Two other general references:
Kantowitz and Sorkin, Human Factors: Understanding People-System Relationships. NY: Wiley, 1983.
Sanders and McCormick, Human Factors in Engineering and Design, 6th ed. NY: McGraw-Hill, 1987.

You have to be looking at a visual message to get it;
audio messages you'll get no matter where you're looking.

2.3 Sounds for particular VR tasks.

Several ideas for VR issues in this section came from the paper:
NCSA Technical report TR032, Human Factors in Virtual Environments for the Visual Analysis of Scientific Data, by M. Pauline Baker and Christopher D. Wickens, 1995. (Available here.)

Searching through a data set.

VR worlds.

Many copies of the same sound.

Localizing sounds.

2.4 Using VSS with the CAVE

Before starting up the whole CAVE library, it makes sense to verify that the run-time environment is intact. Also, note that CAVEExit() actually exits the program, so don't put any code after it. This leads to the following order:
  1. BeginSoundServer()
  2. AUDinit()
  3. CAVEConfigure(), CAVEMalloc(), CAVEInit(), CAVEDisplay()
  4. main loop: AUDupdate()'s
  5. (if needed:) AUDterminate()
  6. EndSoundServer()
  7. CAVEExit()

You may prefer to call BeginSoundServer() after CAVEConfigure(), not before, in order to automatically pick up the environment variable $SOUNDSERVER which the cave configuration files can specify for you. Note carefully how this works:

If you use "-L" to specify directories when linking, it's a good idea to put any VSS-specific directories before CAVE directories. Otherwise you might link with an unintended copy of libsnd.a from a CAVE directory.

3. How do I try out the various sonifications?

  1. Start up VSS.
  2. Change to the directory sonifications.

  3. For continuous sounds, type one of the following commands:
      audpanel continuous_probe.ap     continuous_probe.aud
      audpanel continuous_gurgle.ap    continuous_gurgle.aud
      audpanel continuous_vowel.ap     continuous_vowel.aud
      audpanel continuous_whistle.ap   continuous_whistle.aud
      audpanel continuous_dimple.ap    continuous_dimple.aud
      audpanel continuous_wind.ap      continuous_wind.aud
      audpanel continuous_puttputt.ap  continuous_puttputt.aud
    To adjust the sound, wiggle the sliders of audpanel.
    To reset or end the sound, click RESET or EXIT.
    (You can find the program audpanel in the directory where VSS is installed.)

  4. For triggered sounds, type one of the following commands:
      audpanel triggered_swoop.ap triggered_swoop.aud
      audpanel triggered_beep.ap  triggered_beep.aud
    To adjust the sound, wiggle the sliders as before.
    Either click HIT to trigger sounds one at a time,
    or click SENDING to generate a stream of sounds
    (use the Interval slider to adjust the duration between successive sounds, but don't make the duration too short).
    To reset or end the sound, click RESET or EXIT.

5. How do I add one of these sonifications to my application?

First of all, get the basic VSS structure in place: #include "vssClient.h", BeginSoundServer(), EndSoundServer(), AUDinit(), makefile, .aud file.

5.1 How do I add a beep, swoop, or other controllable triggered sound to my application?

  1. Note the name of the sound's message group.
    Note the sound's number of parameters, their meaning, and their suggested range. Here: These numbers come from the .ap file, the lines called "minValues" and "maxValues".
  2. Find the place in your code where you want the triggered sound to be played.
  3. Add the following function call to your code:
    { float myArray[4] = { ... };
    AUDupdateSimple("Beep", 4, myArray); }

    The array of floats should contain four values corresponding to the sound you want to hear. (If you're using a swoop instead of a beep, the function call would be:
    { float myArray[5] = { ... };
    AUDupdateSimple("Swoop", 5, myArray); }

    and similarly for other triggered sounds.

5.2 How do I add a controllable continuous sound to my application?

  1. Note the names of the message groups needed to start, modify, and end the sound. We use continuous_probe.aud and continuous_probe.ap as an example here:
    "ButtonDown", "WandMove", "ButtonUp" respectively.
  2. Note the parameters of each message group. Typically, the message groups used to start and to modify the sound will have the same parameters (so you don't need to send a WandMove nanoseconds after the ButtonDown). Here:
    These numbers come from the .ap file, the lines called "minValues" and "maxValues" for each message group.
  3. Find the place in your code where you want the continuous sound to start.
  4. Add the following function call to your code:
    AUDupdateSimple("ButtonDown", 3, array_of_three_floats);
    The array of floats should contain three values corresponding to the sound you want to hear.
  5. Repeat these two steps for where you want the sound to stop, and for any places you want to modify the sound.
    If a message group has zero parameters, as with ButtonUp here, use this function call:
    AUDupdateSimple("ButtonUp", 0, NULL);

6. A fancy example including C code: pinball

This is a simple physical model that generates interesting patterns of discrete events and continuous changes of parameters. A ball moves around a playing field with three kinds of obstacles; friction and gravity affect the ball's motion as well. The first kind of obstacle is a simple wall, which reflects the ball in the usual way. The second kind is an accelerator, which increases the ball's speed while maintaining its direction. The third kind is a randomizer, which preserves the ball's net speed but randomly changes its direction.

Run this example by starting VSS, going to the sonifications/pinball directory and typing one of the following: (You might have to type make first to create the file pinball.)
It works best if your terminal window is 80 columns wide and at least 40 columns high.
Press q or the escape key to quit.

Obstacles, friction, and gravity are implemented by the functions constrainmotion() and moveball() in the file pinball.c.
For details on how the playing field works, refer to the manpage for the "curses" library (man curses). This graphics library works on any Unix machine.

When the ball collides with an obstacle, an appropriate sound sample is played using the SampleActor. Well, that's oversimplified. When the ball collides with an obstacle, what really happens is that a variable (fHitBumper, for example) is incremented by some amount (4). Every frame (time step) of the simulation, this variable is decremented by 1. The sound sample is played only on a "rising edge" of this variable -- when it is now nonzero but was zero in the previous frame. We do this to avoid playing several overlapping sound samples in a very short space of time, if the ball hits several of the same kind of obstacle in quick succession. This is computationally more efficient for the sound server, and avoids the danger of hard clipping from playing too many sounds at once. The code for this is the first part of sendaudiodata() in pinball.c.
A better design would use an actual timer instead of just counting frames. The frame rates of other graphics libraries (GLUT, CAVE, etc.) may not be as steady as that of the curses library!

The second part of sendaudiodata() deals with continuous sounds derived from the ball's motion, in particular its absolute speed vBall,its height by and its vertical speed dy. vBall is mapped to three different parameters which are stored in the array speedarray and passed to the message group mSpeed for use by a pair of FmActors in the file 2fm.aud. Similarly, by and dy are mapped to values in the array ballArray which is sent to the message group mBallY. These blocks of code are enclosed in "if (fabs(x-xPrev) > threshhold)" tests. When such a test fails (for example, if x==xPrev), it prevents the corresponding AUDupdateSimple() from being called. This reduces the number of messages being sent to VSS, which gives VSS more time to actually compute sound.

Commenting out the line "#define SOUND" at the beginning will compile pinball.c without its audio code, so it can still run if a sound server is not available for some reason.

Four separate .aud files are provided, to illustrate how different sounds can be made without recompiling your application (or, if you code it right, even without restarting it!). The code in pinball.c sends the same messages to each .aud file.

This .aud file defines three message groups for the sound-sample playback from ball collisions. In each case, the message group defines a SampleActor, preloads a particular sample file into memory, sets its default amplitude, and adds a single message to the message group (to play that sample file, of course).

The "Speed of ball" message group is continuous, not discrete. It uses two notes of an FmActor, slightly detuned. The three parameters *0 *1 *2 map to amplitude, index of modulation, and carrier/modulator ratio respectively, of both notes.

The "Height and vertical speed of ball" message group uses a single FmActor note. The first parameter *0 adjusts the note's carrier frequency to make the high-low swooping sound, and the second parameter *1 adjusts the index of modulation to make the sound more or less penetrating.

This is a slight variant of 2fm.aud. The message groups for ball collisions now use the speed of the ball to adjust the volume of the sound produced. Instead of a simple "SetAmp hB 1;", they use the first argument passed in AUDupdateSimple() as follows: "AddMessage mBumper SetAmp hB *0;".

This .aud file also dispenses with the "dentist's drill" sound produced by the note nBallY. It sounds the nicest of all the .aud files provided with this example.

This stripped-down version plays only a single soundfile for all collisions, and sonifies only the height of the ball.

This maps the ball's height and vertical velocity to the pitch and brightness of an additive-synthesis tone (AddActor). Collisions, instead of triggering soundfiles, cause noise bursts to be played with the NoiseActor. Accelerator collisions actually play a rising-pitch additive-synthesis note. Walls play a short noise burst, decaying at the end. Randomizers play a longer, more irregular noise burst by choosing a small modulation value.

7. How can I "debug" or try out my sounds, without running my whole application?

Use audpanel. This program, together with a .ap file, acts as a stub or simulator of your whole application. Its buttons and sliders let you exercise your .aud file directly. The .ap file acts as the "glue" between the .aud file's message groups and audpanel's sliders.

Your own application doesn't use a .ap file: its AUDupdate...() calls connect directly to the .aud file's message groups.

8. How do I effectively combine several sounds at once?

Playing many sounds at once is nontrivial.
  1. It uses more CPU power. If you exceed what your CPU can provide, you'll hear interruptions in the sound, a crunchy fast stuttering. You can check if VSS's CPU usage is near 100% with a CPU meter (gr_osview on SGI, Task Manager in Windows NT). One cure is to run VSS with fewer channels or at a lower sampling rate.
  2. The total number of sounds may get too loud. You can only get so loud before distortion called "hard clipping" occurs. This sounds like a really bad electric guitar. You can confirm that hard clipping is happening by noticing if the sound returns to normal when you turn down the slider in VSS's control panel (or the volume slider in SGI's "apanel").
    Quick fix: start VSS with the option -soft which replaces the nasty hard clipping with soft clipping.
    Sledgehammer fix: start VSS with the option -limit which prevents hard clipping entirely, no matter how hard you try. The tradeoff is that quieter sounds may become too quiet. (Technical note: this is a "fast-knee slow-release limiter".)
    A more permanent cure is to reduce the number of simultaneous sounds, and/or reduce how loud the individual sounds are. The numbers specifying these things may be in the .aud file, or they may sent from your application via AUDupdate.
Even if everything's playing okay, it may still be hard to hear individual things. There are a few simple orchestration tricks to keep individual sounds intelligible and recognizable.
Advanced exercise: with good headphones, listen to densely engineered post-1985 popular music (especially dance and techno) and you'll hear all of the following methods at work.
  1. If you have at least two speakers, you can use the SetPan command to put different sounds at different positions in the stereo field. (In a 4-speaker setup, the sounds will spread all around like "surround sound.")

    Anywhere in a .aud file where you see BeginSound, you can append the phrase SetPan x to "pan" or move the sound from far left to far right as x varies from -1 to 1. (The term "pan" comes from "pan pot", an abbreviation of the audio engineering term "panoramic potentiometer" which is itself panoramic.)

  2. If you're using the 8-speaker system in the CAVE, the SetXYZ command will let you use distance and elevation as well to distinguish one sound from another. (Or just use both SetPan and SetDistance.) SetXYZ will do the right thing with fewer speakers, too, so you can still hear what's going on with two speakers.

  3. Use different frequency ranges for different sounds. You can do this by ear like Beethoven and Rimsky-Korsakov, or you can see what frequencies your sounds have by starting VSS with the option -graphspectrum. VSS also has other command-line flags which do various things.

  4. One sound at a time. If you have several continuous sounds playing loud all the time, it's like a party with too many drunks. Entertaining but not that informative. But at a good dinner conversation, people take turns. The repartee may still be quick, but it's easy to follow. Non-speech sound is the same: have continuous sounds arrive and depart. Bonus: doing this is a great way to avoid hard clipping.

    A related principle: if a sound isn't changing, it should probably fade into the background.

    Another principle: you can only pay attention to so many sounds at once. Out of all the sounds that might play at a given moment, play only the 5 or 10 most important. This is what videogames do. (You can learn a lot from the better videogames.)

  5. Use different tempos. If you have several layers of continuous sound, impose various tempos (amplitude envelopes or actual rhythms) on different sounds with the Sequence Actor to make it easier to tune in to a particular sound.

    [ up to main page ]

    Back: Parambient sound.