[whatwg] Codecs for <audio> and <video>

[Chris McCormick]

On Wed, Jul 08, 2009 at 09:24:42AM -0700, Charles Pritchard wrote:
> There are two use cases that I think are important: a codec  
> implementation (let's use Vorbis),
> and an accessibility implementation, working with a <canvas> element.

Here are a few more use-cases that many people would consider just as
important:

* Browser based music software and synthesis toys.
* New types of 'algorithmic' music like that pioneered by Brian Eno.
* Browser based games which want to use procedural audio instead of
pre-rendered sound effects.

I'd like to reiterate the previously expressed sentiment that only implementing
pre-rendered audio playback is like having a browser that only supports static
images loaded from the server instead of animations and <canvas> tags.

What is really needed is a DSP vector processor which runs outside of ECMA
script, but with a good API so that the ECMAscripts can talk to it directly.
Examples of reference software, mostly open source, which do this type of thing
follow:

* Csound
* Supercollider
* Pure Data
* Nyquist
* Chuck
* Steinberg VSTs

I am going to use the terms "signal vector", "audio buffer", and "array"
interchangeably below.

Four major types of synthesis would be useful, but they are pretty much
isomorphic, so any one of them could be implemented as a base-line:

* Wavetable (implement vector write/read/lookup operators)
* FM & AM (implement vector + and * operators)
* Subtractive (implement unit delay from which you can build filters)
* Frequency domain (implemnt FFT and back again)

Of these, I feel that wavetable synthesis should be the first type of synthesis
to be implemented, since most of the code for manipulating audio buffers is
already going to be in the browsers and exposing those buffers shouldn't be
hugely difficult. Basically what this would take is ensuring some things about
the audio tag:

* Supports playback of arbitrarily small buffers.
* Seamlessly loops those small buffers.
* Allows read/write access to those buffers from ECMAscript.

Given the above, the other types of synthesis are possible, albeit slowly. For
example, FM & AM synthesis are possible by adding adding/multiplying vectors of
sine data together into a currently looping audio buffer. Subtractive synthesis
is possible by adding delayed versions of the data in the buffer to itself.
Frequency domain synthesis is possible by analysing the data in the buffer with
FFT (and reverse FFT) and writing back new data. I see this API as working as
previously posted, by Charles Prichard, but with the following extra
possibility:

<audio id='mybuffer'>
buffer = document.getElementById("mybuffer");
// here myfunc is a function which will change
// the audio buffer each time the buffer loops
buffer.loopCallback = myfunc;
buffer.loop = True;
buffer.play();

Of course, the ECMA script is probably going to be too slow in the short term,
so moving forward it would be great if there was a library/API which can do the
following vector operations in the background at a speed faster than doing them
directly, element by element inside ECMAscript (a bit like Python's Numeric
module). All inputs and outputs are signal vectors/audio tag buffers:

* + - add two signal vectors (2 input, 1 output)
* * - multiply two signal vectors (2 input, 1 output)
* z - delay a signal vector with customisable sample length (2 input, 1 output)
* read - do a table lookup (1 input, 1 output)
* write - do a table write (2 input, 1 output)
* copy - memcpy a signal vector (1 input, 1 output)
* fft do a fast fourier transform - (1 input, 2 output)
* rfft do a reverse fast fourier transform - (2 inputs, 1 output)

It would be so great if it was possible to unify the above into an API that
looked and worked something like this:

<audio id='mybuffer'>

outbuffer = document.getElementById("mybuffer");

b = new AudioBuffer(64)
for (x=0; x<64; x++)
	b[x] = Math.sin(x / 64 * Math.PI)a;

// inside the loopCallback do a vector multiplication of the data in our buffer
// with a sine wave we created earlier.
outbuffer.multiply(b);

I hope this email is not too obvious and helps clarify thinking rather than
confusing things. As a game developer and music software developer I look
forward to making dynamic audio applications on the web!

Best regards,

Chris.

-------------------
http://mccormick.cx