[whatwg] Binary Streaming with XHR

[Jussi Kalliokoski]

Hi Cyril,

Thanks for your comments. Responses inline.

On Fri, Aug 24, 2012 at 4:49 PM, Cyril Concolato <
cyril.concolato at telecom-paristech.fr> wrote:

> Hi Jussi,
>
> In general, I like the idea of having streaming over XHR. Thanks for
> bringing it up. I have some comments/questions inline.
>
> Le 8/24/2012 12:23 PM, Jussi Kalliokoski a écrit :
>
>  Hello,
>>
>> I've got a little proposal to solve a problem we're facing with one of our
>> codebases ( aurora.js [1], i.e. audio codecs in JavaScript ).
>>
>> We need to stream the audio files (you don't want to store a 2 hour long
>> live stream in the memory or wait until a whole file is loaded before
>> playing). Currently we're using partial requests to fetch the data, but
>> it's very expensive both in terms of CPU (currently the streaming takes
>> more CPU than the actual decoding, I suspect this is a sum of garbage
>> collection and other factors) and network.
>>
> Just to make sure I understand. How is your use case different from
> DASH-JS [1][2] ? These implementations seem to do streaming over XHR
> without too many problems. Are you having problems because you're doing
> audio-only and thus the network processing is very visible? Is it because
> you are using small chunks (for low latency?) and you get too many events?


Both. :) We aren't exactly having problems (although with the current Audio
APIs forcing you to do audio processing in the main thread you might get a
glitch or two while the file is being loaded), but better performance is
always good, yes? ;)


> My proposal is that we add a feature for streaming to XHRs that are of the
>> arraybuffer response type.
>>
>> My first idea was that if you set a property called "chunkSize" on the
>> XHR,
>> it would start dispatching an event called "onchunk" whenever a chunk of
>> that size has been loaded.
>>
>

>  Is it size or media duration or both? The size might be roughly constant
> for audio data but not for other types say video. You might want to receive
> events only when some media duration has been received and not care about
> the size, no?


Not necessarily, since the decoding is done in multiple stages (probing,
demuxing, etc.). And if the streaming provided you with sizes relative to
media duration that would assume the browser should keep track of the media
file as well, which would be problematic if the codec is unsupported by the
browser.

But now that you mention it, it would be useful to have more fine-grained
control over the data flow. I.e. you should be able to set the "chunk"
property again on the fly, to a different sized buffer. This would allow
our codecs to dynamically adjust the chunk sizes based on whether we have
enough data to keep playing for a while and other information the decoder
has.


>  That event would contain a property "chunk" that
>> would be an array buffer containing that data. However, I'm having second
>> thoughts on that approach as it is likely to produce a lot of garbage.
>>
>> Instead, my refined idea is that you'd set a "chunk" property on the XHR
>> to
>> an ArrayBuffer, and each time a chunk of the size of the "chunk" was
>> ready,
>> it would set the values of the "chunk" property accordingly and the
>> "onchunk" event would be dispatched.
>>
> If you want to be able to feed decoders directly (ą la DASH), you might
> want to align with the Media Source Extension API which uses
>
>  voidappend  <http://dvcs.w3.org/hg/html-**media/raw-file/tip/media-**
> source/media-source.html#dom-**append<http://dvcs.w3.org/hg/html-media/raw-file/tip/media-source/media-source.html#dom-append>>(Uint8Array
> data);
>
>
> I think you would need too some timing information about the data you're
> receiving.
>

We're using Audio APIs to play back the data, but thanks for the link,
hadn't come across that document before!

Cheers,
Jussi