The once and future audio engine (notes on a proposed re-design)

[ericrosenbaum]

We will probably need to re-design the audio engine. This is mainly to separate the life cycle of a SoundPlayer from that of a clone. In 2.0, a sound started by a clone continues after the clone has been deleted (see scratchfoundation/scratch-vm#949), so the re-design is required to support that behavior.

@thisandagain and I discussed a possible new design for the audio engine, which I wanted to document here, because this work is likely to be deferred for another few months.

TL;DR: AudioEngine manages a set of persistent SoundPlayers, one for each sound

Requirements

A target (sprite or clone) needs to be able to:

• Play a sound owned by the parent sprite
• Play multiple sounds at once
• Play one one copy of each sound at a time
  • i.e. multiple copies of one sound can not be played simultaneously - it restarts
• Stop all sounds for this target
• Set and change volume for this target
  • by updating a webaudio node
• Set and change pan for this target
  • by updating webaudio nodes
• Set and change pitch effect (i.e. playback rate) for all sounds currently being played by this target
  • Each sound has a buffer source node that must be explicitly updated

The requirements above are currently being met, but this one is not:

• A sound started by a clone plays to completion, even if the clone is deleted while it plays

Current Audio Engine Design

Here's the current (April 2018) setup of the audio engine:

• There's a single Audio Engine, which manages audioBuffers used by all sprites
• Each target (sprite or clone) owns an Audioplayer, which manages audio effects.
• AudioPlayer creates a new SoundPlayer each time a sound is played.
• SoundPlayer is disposed at some point after the sound ends.
• AudioPlayer is disposed when the target is deleted.

A sprite and its clone share a single activeSoundPlayers list, which is used to

• Restart sounds already playing
• Stop all sounds
• Update the pitch effect

Proposed Re-design

The idea is to reduce the system down to just an AudioEngine that manages a set of SoundPlayers. Each sound created in Scratch is associated with a single persistent SoundPlayer object.

The Audio Engine creates and deletes these SoundPlayers to stay in sync with the VM.

Each SoundPlayer would know if it is currently playing, and when played, restart itself if it is already playing.

The Audio Engine can update a SoundPlayer's parameters (volume, pitch, pan) when the sound starts or while it plays.

When a sound is started by a target, the VM would use the parameters for that target to update the SoundPlayer.

Questions

When the VM updates sound parameters for a target (e.g. using a "set volume" block), the AudioEngine will need a way to update those parameters on all the SoundPlayers currently being played by that target. How should this work? Two ideas:

• The targets themselves (in the sound blocks in the VM) could keep track of the SoundPlayers they are currently using to play sounds.
  • It would be a bit complicated to keep this in sync, because one target can restart a sound that another target started, etc.
• Each SoundPlayer could store the ID of the target that most recently played it.
  • In that case, the question is how to efficiently update all the SoundPlayers for a particular target. Most simply, the AudioEngine would need to step through all of its SoundPlayers, checking each for the target ID. If necessary, the AudioEngine could provide a way to cache the results of this search for each target ID in a map, clearing this cache each time a sound is played, stopped or deleted.

The other big question is how this will affect other users of the AudioEngine, such as the sound library. Hopefully the changes to keep things working would be small.

[gnarf]

I believe we closed this with the scratch-audio work but never closed this issue - If I'm wrong reopen?