[whatwg] Codecs for <audio> and <video>

[Chris McCormick]

Hi Boris,

On Sat, Aug 08, 2009 at 02:15:19PM -0400, Boris Zbarsky wrote:
> Chris McCormick wrote:
>> Of course, the ECMA script is probably going to be too slow in the short term,
>> so moving forward it would be great if there was a library/API which can do the
>> following vector operations in the background at a speed faster than doing them
>> directly, element by element inside ECMAscript (a bit like Python's Numeric
>> module). All inputs and outputs are signal vectors/audio tag buffers:
>>
>> * + - add two signal vectors (2 input, 1 output)
>> * * - multiply two signal vectors (2 input, 1 output)
>> * z - delay a signal vector with customisable sample length (2 input, 1 output)
>> * read - do a table lookup (1 input, 1 output)
>> * write - do a table write (2 input, 1 output)
>> * copy - memcpy a signal vector (1 input, 1 output)
>> * fft do a fast fourier transform - (1 input, 2 output)
>> * rfft do a reverse fast fourier transform - (2 inputs, 1 output)
>
> I'm sort of wondering what the performance of these would actually be if  
> implemented directly in ECMAScript, before we decide that's too slow and  
> start looking for alternate solutions.  Do you happen to have any sample  
> implementations?  What size arrays are we talking about here?
>
> I just did a quick test in SpiderMonkey, and adding two arrays of  
> integers with 441000 elements each (so 10s of 44.1kHz audio; the time  
> includes allocating the sum array and all that) element-by-element like 
> so:
>
>   var a3 = new Array(size);
>   for (var j = 0; j < size; ++j) {
>     a3[j] = a1[j] + a2[j];
>   }
>
> takes about 25ms on my computer.  Multiplication takes about 35ms.  
> Duplicating an array takes about 20ms.  This is without any of the  
> in-progress optimizations for type-specializing arrays, etc.
>
> What sort of performance are we looking for here?

It's a bit of an open ended how-long-is-a-piece-of-string sort of a question in
that you will generally make synthesizers which require less CPU than what your
computer is able to provide, for the obvious reason that they won't work
otherwise. So the real answer is that you want the DSP system to go absolutely
as fast as possible I guess, so that you can squeeze as much synthesis out as
possible.

I'll throw some numbers out there anyway. A game with procedural audio, or a
synth, or a piece of algorithmic music might contain between tens and tens of
thousands of such vector operatons per frame, and buffers might be between 10ms
and 100ms = vector sizes of 441 samples to 4410 samples. So you could do some
simple synthesis with pure Javascript if it was able to loop through say 100
arrays of 4410 samples each, doing vector operations on those arrays, in under
100ms.

Hope this helps a little bit.

Chris.

-------------------
http://mccormick.cx