[whatwg] Recording interface (Re: Peer-to-peer communication, video conferencing, and related topics (2))
Stefan Håkansson LK
stefan.lk.hakansson at ericsson.com
Tue Mar 29 23:37:09 PDT 2011
> -----Original Message-----
> From: whatwg-bounces at lists.whatwg.org
> [mailto:whatwg-bounces at lists.whatwg.org] On Behalf Of
> whatwg-request at lists.whatwg.org
> Sent: den 29 mars 2011 20:33
> To: whatwg at lists.whatwg.org
> Subject: whatwg Digest, Vol 84, Issue 69
> >> > I also believe that the recording interface should be
> removed from
> >> > this part of the specification; there should be no requirement
> >> > that all streams be recordable.
> > Recording of streams is needed for some use cases unrelated
> to video
> > conferencing, such as recording messages.
> Having a recording function is needed in multiple use cases;
> I think we all agree on that.
> This is mostly a matter of style, which I'm happy to defer on.
> >> > The streams should be regarded as a control surface,
> not as a data
> >> > channel; in many cases, the question of "what is the
> format of the stream at this point"
> >> > is literally unanswerable; it may be represented as hardware
> >> > states, memory buffers, byte streams, or something
> completely different.
> > Agreed.
> >
> >
> >> > Recording any of these requires much more specification
> than just
> >> > "record here".
> > Could you elaborate on what else needs specifying?
> One thing I remember from an API design talk I viewed:
> "An ability to record to a file means that the file format is
> part of your API."
>
> For instance, for audio recording, it's likely that you want
> control over whether the resulting file is in Ogg Vorbis
> format or in MP3 format; for video, it's likely that you may
> want to specify that it will be stored using the VP8 video
> codec, the Vorbis audio codec and the Matroska container
> format. These desires have to be communicated to the
> underlying audio/video engine, so that the proper transforms
> can be inserted into the processing stream, and I think they
> have to be communicated across this interface; since the
> output of these operations is a blob without any inherent
> type information, the caller has to already know which format
> the media is in.
This is absolutely correct, and it is not only about codecs or
container formats. Maybe you need to supply info like audio
sampling rate, video frame rate, video resolution, ...
There was an input on this already last November:
http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2010-November/029069.html
>
> Clearer?
>
>
>
> >
>
>
>
> ------------------------------
>
> Message: 2
> Date: Tue, 29 Mar 2011 15:27:58 +0200
> From: "Wilhelm Joys Andersen" <wilhelmja at opera.com>
> To: whatwg at lists.whatwg.org
> Subject: [whatwg] <details>, <summary> and styling
> Message-ID: <op.vs3w0wvrm3w6te at kunnskapet.oslo.osa>
> Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes
>
> Hi,
>
> I'm currently writing tests in preparation for Opera's implementation
> of <details> and <summary>. In relation to this, I have a few
> questions
> about issues that, as far as I can tell, are currently
> undefined in the
> specification.
>
> The spec says:
>
> "If there is no child summary element [of the details element], the
> user agent should provide its own legend (e.g. "Details")." [1]
>
> How exactly should this legend be provided? Should the user agent add
> an implied <summary> element to the DOM, similar to <tbody>, a
> pseudo-element, or a magic non-element behaving differently from both
> of the above? In the current WebKit implementation[2], the UA-provided
> legend behaves inconsistently from from an author-provided <summary>
> in the following ways:
>
> * Although it can be styled with rules applying to
> <summary>, it does
> not respond to :hover or :first-child.
>
> * With regards to text selection, it behaves more like an <input
> type='submit'> than a user-provided <summary>. Text within this
> implied element may only be selected _together_ with the text
> preceding and following it.
>
> * A different mouse cursor is used.
>
> This indicates that it is slightly more magic than I would prefer. I
> believe a closer resemblance to an ordinary element would be more
> convenient for authors - a ::summary pseudo element with "Details" as
> its content() might be the cleanest approach, although that would
> require a few more bytes in the author's stylesheet to cater to both
> author- and UA-defined summaries:
>
> summary, ::summary {
> color: green;
> }
>
> Furthermore, the rendering spec says:
>
> "The first container is expected to contain at least one line box,
> and that line box is expected to contain a disclosure
> widget (typically
> a triangle), horizontally positioned within the left padding of the
> details element." [3]
>
> For user agents aiming to support the suggested default rendering, how
> should the disclosure widget be embedded? Ideally, graphical browsers
> should all do this in a similar manner, and in a way that
> allows authors
> to style these elements to the same extent as any other element.
>
> There are several options:
>
> * A ::marker pseudo element[4].
> * A default, non-repeating background image positioned within
> the recommended 40 pixel left padding.
> * A method similar to list-style-type or list-style-image.
> * A magically embedded, unstylable widget.
>
> I prefer the first, if possible. The default rendering could
> be something
> like:
>
> details summary::marker {
> content: "?";
> color: black;
> }
>
> details[open] summary::marker {
> content: "?";
> }
>
> RTL might add some additional complexity here, however.
>
> The spec also says:
>
> "The user agent should allow the user to request that the details
> be shown or hidden."[5]
>
> Given, still, a user agent aiming to support the suggested default
> rendering, with a pointing device (mouse, touch) available,
> which parts
> of the <details> or <summary> element should be clickable?
>
> The spec only says:
>
> "[The disclosure] widget is expected to allow the user to
> request that
> the details be shown or hidden.
>
> That's a rather small clickable area, which might get
> troublesome to hit
> on a fuzzy touchscreen or for someone with limited motor
> skills. I suggest
> the whole block area of <summary>, too, is made clickable -
> as if it was
> a <label> for the ::marker.
>
> The behaviour for a JavaScript click() on the <details> and <summary>
> elements should also be defined.
>
>
> Whether or not the answers to the questions above should be
> part of the
> spec or not, I do not know. But browser vendors should try to come to
> some sort of a consensus here, to limit the pain of
> style-minded authors.
>
>
> [1] :
> http://www.whatwg.org/specs/web-apps/current-work/multipage/in
> teractive-elements.html#the-details-element
> [2] : Tested in Google Chrome 12.0.712.0
> [3] :
> http://www.whatwg.org/specs/web-apps/current-work/multipage/re
> ndering.html#the-details-element-0
> [4] : http://www.w3.org/TR/css3-lists/#markers
> [5] :
> http://www.whatwg.org/specs/web-apps/current-work/multipage/in
> teractive-elements.html#the-details-element
>
> --
> Wilhelm Joys Andersen
> Core, Opera Software
>
>
> ------------------------------
>
> Message: 3
> Date: Tue, 29 Mar 2011 15:35:29 +0200
> From: Stefan H?kansson LK <stefan.lk.hakansson at ericsson.com>
> To: Ian Hickson <ian at hixie.ch>, whatwg <whatwg at whatwg.org>
> Subject: [whatwg] Media negotiation (RE: Peer-to-peer communication,
> video conferencing, and related topics (2))
> Message-ID:
>
> <BBF498F2D030E84AB1179E24D1AC41D612066B0DC7 at ESESSCMS0362.eemea
> .ericsson.se>
>
> Content-Type: text/plain; charset="iso-8859-1"
>
> > > > > !The web application must be able to !If the video
> > is going to be displayed !
> > > > > !define the media format to be used for !in a large
> > window, use higher bit- !
> > > > > !the streams sent to a peer.
> > !rate/resolution. Should media settings!
> > > > > ! !be allowed to
> > be changed during a !
> > > > > ! !session (at
> > e.g. window resize)? !
> > > >
> > > > Shouldn't this be automatic and renegotiated
> dynamically via SDP
> > > > offer/answer?
> > >
> > > Yes, this should be (re)negotiated via SDP, but what is
> > unclear is how
> > > the SDP is populated based on the application's preferences.
> >
> > Why would the Web application have any say on this? Surely
> > the user agent
> > is in a better position to know what to negotiate, since it
> > will be doing
> > the encoding and decoding itself.
> The best format of the coded media being streamed from UA a
> to UA b depends on a lot of factors. An obvious one is that
> the codec used is supported by both UAs.... As you say much
> of it can be handled without any involvement from the application.
>
> But let's say that the app in UA a does "addStream". The
> application in UA b (the same application as in UA a) has two
> <video> elements, one using a large display size, one using a
> small size. The UAs don't know in which element the stream
> will be rendered at this stage (that will be known first when
> the app in UA b connects the stream to one of the elements at
> "onaddstream"), so I don't understand how the UAs can select
> a suitable video resolution without the application giving
> some input. (Once the stream is being rendered in an element
> the situation is different - then UA b has knowledge about
> the rendering and could somehow inform UA a.)
>
> Stefan
>
> ------------------------------
>
> Message: 4
> Date: Tue, 29 Mar 2011 15:54:32 +0200
> From: James Graham <jgraham at opera.com>
> To: whatwg at lists.whatwg.org
> Subject: Re: [whatwg] <details>, <summary> and styling
> Message-ID: <4D91E498.70002 at opera.com>
> Content-Type: text/plain; charset=UTF-8; format=flowed
>
> On 03/29/2011 03:27 PM, Wilhelm Joys Andersen wrote:
> > Hi,
> >
> > I'm currently writing tests in preparation for Opera's
> implementation
> > of <details> and <summary>. In relation to this, I have a
> few questions
> > about issues that, as far as I can tell, are currently
> undefined in the
> > specification.
> >
> > The spec says:
> >
> > "If there is no child summary element [of the details element], the
> > user agent should provide its own legend (e.g. "Details")." [1]
> >
> > How exactly should this legend be provided? Should the user
> agent add
> > an implied <summary> element to the DOM, similar to <tbody>, a
> > pseudo-element, or a magic non-element behaving differently
> from both
> > of the above?
>
> FWIW I think that, from a spec point of view, it should just
> act as if
> the first block box container in the shadow tree contained some
> UA-provided text i.e. no magic parser behavior.
>
> > This indicates that it is slightly more magic than I would prefer. I
> > believe a closer resemblance to an ordinary element would be more
> > convenient for authors - a ::summary pseudo element with
> "Details" as
> > its content() might be the cleanest approach, although that would
> > require a few more bytes in the author's stylesheet to cater to both
> > author- and UA-defined summaries:
> >
> > summary, ::summary {
> > color: green;
> > }
>
> ::summary could be defined to just match the first block box
> element in
> the <details> shadow tree. That way you could just write
>
> ::summary {color:green}
>
> for both cases. I note that optimising for the non-conforming
> case seems
> a bit unnecessary, however.
>
> > That's a rather small clickable area, which might get
> troublesome to hit
> > on a fuzzy touchscreen or for someone with limited motor
> skills. I suggest
> > the whole block area of <summary>, too, is made clickable -
> as if it was
> > a <label> for the ::marker.
>
> Making the whole ::summary clickable would seem consistent
> with the rest
> of the platform where labels are typically clickable.
>
>
> ------------------------------
>
> Message: 5
> Date: Tue, 29 Mar 2011 10:34:24 -0700
> From: Adam Barth <w3c at adambarth.com>
> To: whatwg <whatwg at lists.whatwg.org>
> Subject: [whatwg] WebSockets and redirects
> Message-ID: <BANLkTinNbj5kb-vA0B7NRos8Yaq0dpb=og at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> In this HyBi thread:
>
> http://www.ietf.org/mail-archive/web/hybi/current/msg06951.html
>
> Folks are arguing that the WebSocket protocol should support HTTP
> redirects during the handshake and that dealing with the security
> consequences of redirects should be dealt with at the API layer.
> If/when that occurs, we should update the API layer to deal with the
> security consequences of the WebSocket protocol following HTTP
> redirects, preferably by aborting any WebSocket connections that the
> server attempts to redirect.
>
> Adam
>
>
> ------------------------------
>
> Message: 6
> Date: Tue, 29 Mar 2011 17:50:52 +0000 (UTC)
> From: Ian Hickson <ian at hixie.ch>
> To: Matthew Kaufman <matthew at matthew.at>
> Cc: whatwg <whatwg at whatwg.org>
> Subject: Re: [whatwg] Peer-to-peer communication, video conferencing,
> and related topics (2)
> Message-ID:
> <Pine.LNX.4.64.1103291739590.25791 at ps20323.dreamhostps.com>
> Content-Type: TEXT/PLAIN; charset=US-ASCII
>
> On Tue, 29 Mar 2011, Matthew Kaufman wrote:
> > On Mar 29, 2011, at 3:00 AM, Ian Hickson <ian at hixie.ch> wrote:
> > > On Thu, 24 Mar 2011, Matthew Kaufman wrote:
> > >>
> > >> That goal is incompatible with legacy interoperability.
> > >
> > > There is no legacy when it comes to UDP data media
> streams. This is a
> > > new protocol, no existing servers implement it.
> >
> > Disagree. IF we specify that the media is sent using
> RTP/SRTP (noting
> > the requirement for a handshake to ensure a willing recipient) then
> > quite a few existing endpoints (SIP phones on desks, PSTN
> gateways) can
> > talk directly to/from browsers. This is a significant advantage.
>
> Sure. The spec currently doesn't say what happens with the
> audio/video
> media at all; if browsers implement the same media transports
> as existing
> legacy devices then we get interop essentially for free (assuming a
> gateway for the signaling channel).
>
>
> > If instead the media is wrapped in (for instance) a masking
> layer, this
> > chance for direct peer-to-peer interoperability is lost.
>
> The masking layer isn't used for the audio/video data. It's
> just for the
> game data, which is a new protocol.
>
>
> > >> In particular, there are significant security advantages to
> > >> end-to-end keying rather than transmitting keys over the
> signaling
> > >> channel.
> > >
> > > Could you elaborate on these?
> >
> > End-to-end keying (see my earlier refs to the DTLS-SRTP
> RFCs) gives you
> > significantly greater privacy for the media (because the
> web service
> > doesn't know the negotiated key), including the possibility
> for perfect
> > forward secrecy for these channels.
>
> For audio/video media, sure (though note that the Web page
> has access to
> record the data on either end, so the privacy is limited to
> whether the
> UA allows the user to control whether recording works or
> not). But for the
> data UDP media channel there is no privacy gain here, since
> all the data
> is generated by the Web service in the first place.
>
>
> > DTLS also builds in existing, vetted security mechanisms instead of
> > rolling something new from scratch.
>
> We have to balance the benefit of having something really
> simple against
> the benefits of having something already vetted. Also, it is
> dangerous to
> assume that something already vetted is automatically fine in a Web
> browser environment. For example, most existing protocols are
> vetted on
> the assumption that the local client is trusted, sicne if the
> local client
> wasn't trusted it could send out any arbitrary packets it
> wanted over the
> network. This assumption doesn't hold in a Web browser
> environment, where
> the local client is split into two parts, one of which is trusted and
> implements the protocol, but the other of which can be
> actively hostile
> and provides all the inputs to the protocol.
>
> With the data UDP media stream protocol, the security aspects are
> reasonably easy to reason about, so it should be straight-forward to
> evaluate even though it is a new formulation. Also, it reuses
> existing
> well-understood techniques in a straight-forward way, which
> also helps.
>
>
> > I have previously shown why an optional API for forcing
> keys would be
> > advantageous. (legacy interop with SDES-keyed SRTP, and
> some multi-party
> > cases)
>
> I apparently missed that; if you could point me to your
> earlier mail or
> reiterate the salient points for this that would be great.
>
> --
> Ian Hickson U+1047E
> )\._.,--....,'``. fL
> http://ln.hixie.ch/ U+263A /, _.. \
> _\ ;`._ ,.
> Things that are impossible just take longer.
> `._.-(,_..'--(,_..'`-.;.'
>
>
> ------------------------------
>
> Message: 7
> Date: Tue, 29 Mar 2011 14:19:34 -0400
> From: Aryeh Gregor <Simetrical+w3c at gmail.com>
> To: Boris Zbarsky <bzbarsky at mit.edu>
> Cc: whatwg at lists.whatwg.org
> Subject: Re: [whatwg] DOM Range: redefining behavior under DOM
> mutation
> Message-ID:
> <AANLkTik83MrFZNn+1fZC7FkrtCr7GdJ_W7OoBuhUzZ5R at mail.gmail.com>
> Content-Type: text/plain; charset=UTF-8
>
> On Mon, Mar 28, 2011 at 8:05 PM, Boris Zbarsky
> <bzbarsky at mit.edu> wrote:
> > One problem here is that there is not a concept of "moved"
> in the DOM.
> > There's just removed and inserted. ?Trying to insert
> something that already
> > has a parent will remove it and then do the insert. ?Thanks
> to the wonders
> > of mutation events and userdata, script can execute in
> state when the node
> > being moved is not in the DOM, which means that ranges need
> to update
> > separately for the removal and insertion because the state
> of the range in
> > the intermediate state with the node outside the DOM needs to be
> > well-defined.
>
> Hmm, I see. I was assuming that insertBefore() would just atomically
> change the parent from the new one to the old one. Looking more
> closely at DOM Core, I see it's specced the way you describe it
> (although nothing says when the mutation events fire). That's
> certainly a problem for my plan. It would be possible to work around
> it by requiring that insertBefore() and similar methods do special
> magic for Ranges independent of the actual DOM mutations done,
> however. That already has to happen for
> insertData()/deleteData()/appendData(), right? All browsers treat
> those differently from just setting the data to the equivalent string.
>
> > Now if we dropped support for mutation events and userdata handlers
> > first.....
>
> Is that feasible? I get the impression implementers would all love
> it, but somehow they haven't done it yet.
>
> > Gecko's implementation of A.insertBefore(B, C), in
> pseudocode and ignoring
> > all the sanity-checking and for cases when B is not a
> document fragment is:
> >
> > ?if (B.ownerDocument != A.ownerDocument) {
> > ? ?A.ownerDocument.adoptNode(B); ?// This can run arbitrary script
> > ?}
> >
> > ?if (B.parentNode) {
> > ? ?B.parentNode.removeChild(B); ? // This can run arbitrary script
> > ?}
> >
> > ?// Mutate the child storage of A to put B in the child list; notify
> > ?// observers
> >
> > Both the removeChild call and the mutation of A's child
> list notify mutation
> > observers. ?Ranges are mutation observers, and use the
> notifications to
> > update themselves. ?So from the point of view of a range,
> the removal and
> > insertion are two distinct events.
> >
> > -Boris
> >
> > P.S. ?I suspect that actual interoperability for the edge
> cases of this
> > stuff is poor: nothing defines whether the adopt happens before the
> > removeChild above, nothing defines what happens if user
> data handlers or
> > mutation events mutate the DOM, etc.
>
> DOM Core says it's supposed to be basically
>
> if (B.ownerDocument != A.ownerDocument) {
> A.ownerDocument.adoptNode(B);
> }
>
> // Insert into the children
> <http://dvcs.w3.org/hg/domcore/raw-file/tip/Overview.html#dom-
> node-insertbefore>
>
> where adoptNode() removes the node from its parent before changing the
> ownerDocument. This seems to be how both IE, Gecko and Opera actually
> behave -- moving a node from one document to another has the
> ownerDocument equal to the old document on DOMNodeRemoved, and the new
> document on DOMNodeInserted. (WebKit throws if you do insertBefore()
> cross-document, and I can't get it to fire any DOM mutation events
> even if it's same-document.) When I thoughtlessly mutated the DOM
> from the event handler, though, I definitely hit a lack of interop.
>
>
> Anyway, then what does Gecko do for execCommand()? Does that just
> have special-case logic to adjust the selection to fit the new DOM?
> It would be nice if we could eliminate that in a clean way, from both
> the specs and implementations.
>
>
> ------------------------------
>
> _______________________________________________
> whatwg mailing list
> whatwg at lists.whatwg.org
> http://lists.whatwg.org/listinfo.cgi/whatwg-whatwg.org
>
>
> End of whatwg Digest, Vol 84, Issue 69
> **************************************
>
More information about the whatwg
mailing list