[whatwg] Stream API Feedback

[Olli Pettay]

On 03/16/2011 05:36 PM, Lachlan Hunt wrote:
> On 2011-03-15 21:58, Robert O'Callahan wrote:
>> Instead of creating new state signalling and control API for streams,
>> what
>> about the alternative approach of letting<video> and<audio> use
>> sensors as
>> sources, and a way to connect the output of<video> and<audio> to
>> encoders?
>
> I'm not entirely sure I understand your proposal, but are you suggesting
> that the input streams from the camera/microphone would first go to
> <video> and <audio> elements, allowing the existing HTMLMediaElement API
> on those elements to be used to control those streams, the output of
> which can then be encoded and recorded to a file or streamed remotely?

I think roc did suggest that.
Perhaps navigator.getUserMedia("audio,video", success, error);
could return an url to the device in the success callback, and that url
could be then set to video.src.


>
> I'm not so sure that would be ideal. The state machinary, assuming you
> mean the networkState, readyState and their associated constants, are
> clearly designed and optimised for obtaining media over a network and do
> not map so well to obtaining streams directly from local devices.
I'd guess reading from local devices would be very similar to reading
video data from local files (which browsers already support).


>
> Many other properties, such as duration, playbackRate, etc. also do not
> have much meaning in the context of streaming media. Some, like
> currentTime, only have limited applicability to streams as it can tell
> you how long its played for, but must be effectively readonly as seeking
> is not possible.
Well, that is the case already with streamed video.



> But that's not particularly useful for the audio element. It's rare that
> the user would want their microphone input to be echoed back to them via
> an audio element. In most cases, when a microphone stream is input into
> an audio element, the audio element itself would need to be muted to
> prevent unwanted and annoying echo or, worse, feedback loops. That would
> only be useful if the audio data were being analysed and output, for
> example, to an audio spectrum visualisation (like with Mozilla's
> experimental audio data API).
Audio (and video) data could be modified before encoding and
streaming it using PeerConnection. That way one could for example
reduce background noise from the audio stream, or 'crop' the video
before sending it. Or if the camera doesn't support grayscale,
the web page could convert the colorful video to grayscale in
order to save network bandwidth.



-Olli