Brian, excellent questions! We will be applying all those types of sounds, and more! In some cases, we can rely on sounds that are clearly, and perhaps universally, associated with a particular concept. The kinds of sounds used to represent, for example, a successful completion of an exercise, or successfully dropping a proton into the center of an atom...we can be confident that there is common agreement on those kinds of sounds. We can leverage sound design, foley artistry, and shared experiences for much of that.

When it comes to more symbolic relationships, such as when we use sound to represent the temperature or velocity or mass of something in a simulation, we start with a foundation of years of research in the auditory display community to determine "population stereotypes" for sound-to-concept mappings. For instance, we know that increasing pitch is a good way to represent increasing speed, urgency, and vertical position on the screen. On the other hand, to represent increasing mass, we use *decreasing* pitch.

We also know that in some cases increasing pitch best represents increasing temperature, but not always. The context, and the listeners, matter a great deal in those cases.

So, we involve the listeners directly in our design process. To the extent possible, we will be employing population stereotypes; where those don't necessarily exist, we will attempt to determine best-designs in systematic ways (i.e., interacting with plenty of students!). And in some cases, we will simply have to make a design decision about how to represent a concept, based on the pedagogical goals of the sim, and our experience. Then we will do our best to make the "design language" we use consistent with and across the many PhET simulations, and congruent with the visual design language already in use. 

These common design languages, or design conventions, will be clearly documented and consistently applied. This will allow us to train the teachers what to expect, and how to share this with their students.

Michael, thanks for your interest and your questions. We know that in some cases there is a good way to convey a certain concept using sounds. For instance, in teh "Build An Atom" simulation, the student picks protons, neutrons, or electrons up from "baskets" and drops them-one at a time- into the center of a target, which represents a nucleus and orbits. As the first proton is dropped, creating Hydrogen, we can play a sound representing the mass of the atom (one proton). As a second proton is added to the nucleus, the mass of the atom doubles, creating Helium. The sound of Helium should seem "heavier" than the sound for Hydrogen. We represent this using a lower pitch. As protons are added, the sound of the resulting element continues to drop in pitch. But what do we do when you have one proton (Hydrogen), and you add one neutron? The element stays the same, but the mass increases. Do we change the pitch, or do something else? We look at the pedagicical goals, and decide that element number is going to be represented by pitch, as described above, but adding neutrons will be represented by reverb. So adding a neutron makes the sound seem "fuller", representing more mass, but not lower pitch. 

So it is a design challenge, and we tackle it with plenty of expertise; but there is also plenty of need to come up with *new* design decisions, since oftentimes we are sonifying things that no one has likely sonified before.

Once we come up with a design framework, we prototype the sounds, plug them into the simulations, and  engage sim designers, teachers, and students in extensive iterative design sessions to see how our design decisions hold up. We evaluate comprehesion, performance, and preference, and factor it all in as we move forward.

When we find sound design approaches that work, be it by good initial design, or iterative refinement, or even good luck, we document that, and re-use it as much as is appropriate.

Hi Janet! I hope my responses to earlier questions shed some light.

I'll also add here that we not only have to consider the "message" we are conveying (e.g., "you successfully added a proton to the nucleus"), but we also have to consider the tone and the "feel" of the sound. The PhET simulations carefully and consistently use a visual look and feel that is playful yet not childish; clean but not cold; kind of cartoon-realistic but not photo-realistic...And our sounds have to fit into that ecosystem. Thus, beyond just sorting out whether to change pitch  to represent mass or not, we also have to think about the aesthetics of the sounds, and whether they match what the visual interface is portraying...even if the users happen to be blind and cannot see the visual interface. After all, they may very well be in a class with a mixture of sighted, low-vision, and totally blind students, and they all need to have a coordinated, consistent, and shared multimodal learning experience.

Glad you are thinking about all this. My one suggestion: It would be great to think about the sounds you are creating as laying the groundwork for others to also sonify phenomena. That is, try for consistency across simulations in the sounds that are used so learners come to predict what sounds mean and so that other sonifiers will implement consistently with what you guys are doing.

Janet

Absolutely! We are definitely working towards consistency across the many PhET sims, but also making sonification strategies that make sense in other, non-PhET domains. We already do that in my lab, but this project is a great opportunity to have broader impact in the larger field of multimodal interfaces!

Chris, we are producing design documents for the sounds, which complement the design documents that exist or are being developed for the visuals, and for the (text) descriptions that are being put in place to support users of screen readers. Thus, a whole library of multimodal design documents will be an important outcome. Annotated examples will also be a big part of the dissemination. After all, you can interact with the simulation, move some protons around, and immediately get a sense of how the sounds support the experience.

Finally, since the technology used to play the sounds is embedded into the same HTML5 code that is used to change the visual interface, and which also supports the various input modalities, the logic of how events trigger sounds is captured in code. This makes it very easy to understand what sounds wil be played, when, why, and how.