[whatwg] Web-based dynamic audio apps - WAS: Re: video tag : loop for ever

[Silvia Pfeiffer]

On Fri, Oct 17, 2008 at 9:38 PM, Dr. Markus Walther <walther at svox.com> wrote:
>
> Eric Carlson wrote:
>>>
> You can't do responsive audio manipulation in the browser without fast,
> low-latency client-side computation. All the server-side proposals miss
> this crucial point.

It's not about one or the other. It's about having functionality that
allows download of just a segment of audio/video from an origin server
to the UA, *and* about the ability to manipulate audio/video locally
on the client. The latter one is not so difficult - you can do a lot
of things in javascript. The first one is the one that requires setup
of standard URIs and mechanisms to support Web proxies with such media
fragments etc.


> For another use case, consider web-based tools for DJs, for mixing and
> combining audio clips. There's a lot of clips on the web. But if
> manipulating them is not realtime enough, people won't care.

Your biggest problem with this use case are compressed files and the
multitude of codec formats that will need to be supported. If think if
you are really keen for such an application on the web, you would
fully download your audio clips and fully decode them to wav and then
apply your manipulations. The manipulations can then be done in
realtime client-side - no further interaction with the Web and no
further interactions with codecs are necessary.

> Robert O'Callahan wrote:
>> On Fri, Oct 17, 2008 at 5:24 AM, Dr. Markus Walther <walther at svox.com
>> <mailto:walther at svox.com>> wrote:
>>
>>     Imagine e.g. an audio editor in a browser and the task "play this
>>     selection of the oscillogram"...
>>
>>     Why should such use cases be left to the Flash 10 crowd
>>
> (http://www.adobe.com/devnet/flash/articles/dynamic_sound_generation.html)?
>>
>>
>> If people go in that direction they won't be using cue ranges etc,
>> they'll be using dynamic audio generation, which deserves its own API.
>
> And I proposed the beginnings of such an API in several postings on this
> list under the topic 'audio canvas', but it seemingly met with little
> interest. Now Flash 10 has some of the things I proposed... maybe that's
> a louder voice?

May be a bit early for that. Let's get audio and video into the Web first.


> Silvia Pfeiffer wrote:
>>
>> Linking to a specific time point or section in a media file is not
>> something that needs to be solved by HTML. It is in fact a URI issue
>> and is being developed by the W3C Media Fragments working group.
>>
>> If you use a URI such as http://example.com/mediafile.ogv#time=12-30
>> in the src attribute of the video element, you will not even have to
>> worry about "start" and "end" attributes for the video element.
>
> Unless Media Fragments can be a) set dynamically for an already
> downloaded media file _without triggering re-download_,  b) time
> specification can be accurate to the individual sample for the case of
> audio, c) W3C finishes this quickly enough and d) browsers take the W3C
> recommendation seriously, it is not an alternative for my use cases.

a) is about choice. If you fully download a video, then you can most
certainly do all sorts of local dynamic manipulation of it. The use
case of remote media fragments is e.g. for search, where you are
looking for a particular segment out of a long file (e.g. 8 hours of
parliamentary sitting recorded) and don't want to wait until the full
file has downloaded to just play the bit you are interested in. Your
use cases don't fit that need - initial download time is irrelevant to
you, but interactivity is of importance.

b) time specifications will never give you sample accuracy for a
compressed codec. A compressed codec can only really address based on
encoding blocks. For your needs, you will probably always have to
decode your audio to a raw format to do the rich manipulations you are
after.

c) a temporal media fragment addressing scheme is due within a year
http://www.w3.org/2008/01/media-fragments-wg.html .

d) it's not only the browser vendors that have to do work - it's the
media formats that will need to write client-side support for media
fragment delivery that have the most work to do.


Cheers,
Silvia.