[whatwg] Peer-to-peer : comments from game perspective

[Erik Moller]

Ok, firstly let me just say that I'm thrilled about the peer to peer 
stuff moving forward, great work Ian. I wish I had a little more time to 
get stuck into this now and read the specs a bit more thoroughly, but 
I'll try to reply to all your comments that I generated. (Amazingly 
ö still seems to be causing troubles in 2011.)
This is just from my very narrow browser-game perspective, so don't read 
anything into it about video conferencing.. I choose to ignore that.

On 3/18/11 5:45 AM, Ian Hickson wrote:
> On Tue, 4 May 2010, Erik M�ller wrote:
>> I'm an old gamedev recently turned browserdev so this is of particular
>> interest to me, especially as I'm currently working on WebSockets.
>> WebSockets is a nice step towards multiplayer games in browsers and will
>> be even better once binary frames are speced out but as Mark says
>> (depending on the nature of the game) gamedevs are most likely going to
>> want to make their own UDP based protocol (in client-server models as
>> well). Has there been any discussions on how this would fit under
>> WebSockets?
> There has not, as far as I'm aware.
>
> PeerConnection could be used by a server as well, of course.
>
>
I agree it doesn't make sense to try to cram anything more into 
WebSockets... I think it was just me and Mark F. that were eager to get 
things moving on the peer to peer side and WebSockets seemed like the 
only thing that had a bit of momentum back then.

> On Tue, 1 Jun 2010, Erik M�ller wrote:
>> The majority of the on-line games of today use a client/server model
>> over UDP and we should try to give game developers the tools they
>> require to create browser based games. For many simpler games a TCP
>> based protocol is exactly what's needed but for most real-time games a
>> UDP based protocol is a requirement. Games typically send small updates
>> to its server at 20-30Hz over UDP and can with the help of entity
>> interpolation and if required entity extrapolation cope well with
>> intermittent packet loss.
> Does PeerConnection address this use case to your satisfaction?
>
> Note that currently it does not support binary data, but I've built in an
> extension mechanism to make this easy to add in the future.
>
It is looking very promising at least. I won't say yes because I know 
there will always be things missing once you start using it in the real 
world. I guess doing some extra investigation whether those additional 
20 (?) bytes per packet are really necessary would be good. I'll have to 
leave that to someone with more expertise in that area though.
>
> On Wed, 2 Jun 2010, Erik M�ller wrote:
>> No it can't be UDP, it'll have to be something layered on top of UDP.
>> One of the game guys I spoke to last night said "Honestly, I wish we
>> just had real sockets.  It always seems like web coding comes down to
>> reinventing a very old wheel in a far less convenient or efficient
>> manner." To some extent I agree with him, but there's the security
>> aspect we have to take into account or we'll see someone hacking the CNN
>> website and injecting a little javascript and we'll have the DDOS attack
>> of the century on our hands.
> For the data UDP media stream in PeerConnection I tried to make it as pure
> UDP as I could, while still being safe and still being extensible. The
> packets are (doubly) obfuscated to prevent cross-protocol attacks, and you
> can only send data to an end-point that negotiated a key via SDP
> offer/answer and participated in ICE to select how the packets are routed,
> but beyond that it's as raw as I could make it. Hopefully it's enough.
>
It is looking good.
>> The reason I put down "Socket is bound to one address", "Reliable
>> handshake", "Reliable close handshake" and "Sockets open sequentially"
>> was for that exact reason, to try to make it "DOS and tamper safe". The
>> "Sockets open sequentially" means that if you allocate two sockets to
>> the same server the second socket will wait for the first one to
>> complete its handshake before attempting to connect.
> I haven't done this, but since the other server has to participate in the
> ICE processing, and can delay the start of that indefinitely, it seems
> that we're safe here.
>
Agreed.
> On Thu, 3 Jun 2010, Erik M�ller wrote:
>> On Wed, 02 Jun 2010 19:48:05 +0200, Philip Taylor wrote:
>>> So they seem to suggest things like:
>>> - many games need a combination of reliable and unreliable-ordered and
>>> unreliable-unordered messages.
>> One thing to remember here is that browsers have other means for
>> communication as well. I'm not saying we shouldn't support reliable
>> messages over UDP, but just pointing out the option. I believe for
>> example World of Warcraft uses this strategy and sends reliable traffic
>> over TCP while movement and other real-time data goes over UDP.
> That would indeed make sense.
>
>
>>> - many games need to send large messages (so the libraries do
>>> automatic fragmentation).
>> Again, this is probably because games have no other means of
>> communication than the NW-library. I'd think these large reliable
>> messages would mostly be files that need to be transferred
>> asynchronously for which browsers already have the tried and tested
>> XMLHttpRequest.
> Are the large messages always reliable messages?
>
I can of course only speak from my experience from the games I've worked 
on, but these large messages have typically been updated content. 
Textures, level data etc. More recently using a bit torrent system has 
been popular for distributing updated content. So, yeah, those large 
messages have always been reliable.
>>> - many games need to efficiently send tiny messages (so the libraries
>>> do automatic aggregation).
>> This is probably true for many other use-cases than games, but at least
>> in my experience games typically use a bit-packer or range-coder to
>> build the complete packet that needs to be sent. But again, it's a
>> matter of what level you want to place the interface.
> This seems relatively easy to layer on top of the current protocol in the
> spec, but if we find it commonly used we can also add it explicitly as an
> extension.
>
I'd suggest just keeping the API as simple as possible. With JavaScript 
kicking ass and taking names in terms of performance the last couple of 
years it seems less necessary to build things like that into the API. 
Besides, it seems every NW-engineer have their own favourite bitpacker.
>>> Perhaps also:
>>> - Cap or dynamic limit on bandwidth (you don't want a single web page
>>> flooding the user's network connection and starving all the TCP
>>> connections)
> Not really sure what the spec should say about this.
>
>
>>> - Protection against session hijacking
>> Great
> The spec uses an encryption mechanism to prevent this.
>
>
>>> - Protection against an attacker initiating a legitimate socket with a
>>> user and then redirecting it (with some kind of IP (un)hijacking) to a
>>> service behind the user's firewall (which isn't a problem when using
>>> TCP since the service will ignore packets when it hasn't done the TCP
>>> handshake; but UDP services might respond to a single packet from the
>>> middle of a websocket stream, so every single packet will have to be
>>> careful not to be misinterpreted dangerously by unsuspecting
>>> services).
> The packets are masked so that you couldn't do anything but DOS attacks in
> this kind of scenario. (And you can do those already with TCP.)
>
> On Thu, 10 Jun 2010, Erik M�ller wrote:
>> As discussed the following features/limitations are suggested: -Same API
>> as WebSockets
> I don't see how that would work. I've made them as similar as possible,
> but I don't think it makes sense to go further.
>
>
Agreed.
>> with the possible addition of an attribute that allows the
>> application developer to find the path MTU of a connected socket.
> What's the use case?
>
The use case is simply that intermediaries can have different MTUs and 
exceeding those may cause them to just unconditionally drop the packets. 
I haven't verified this recently though, that's just the way it used to 
be in the days...
>> -Max allowed send size is 65,507 bytes.
> Currently 65470, to handle the various headers used (see the spec).
>
>
>> -Socket is bound to one remote address at creation and stays connected
>> to that host for the duration of its lifetime.
> I've specced it in such a way that ICE could rebind the connection later;
> is that ok?
>
I can't remember why I wrote that, but I assume it had something with 
security to do, but It's not my area of expertise so feel free to ignore 
that.
>> -IP Broadcast/Multicast addresses are not valid remote addresses and
>> only a set range of ports are valid.
> I've left this up to the ICE layer.
>
>
>> -Reliable handshake with origin info (Connection timeout will trigger
>> close event.)
> Not sure what the handshake should do here. Could you elaborate?
>
> Also there's currently no origin protection for peer-to-peer stuff (there
> is for the STUN/TURN part; the origin is the long-term credential). We
> could certainly add something; how should it work? What are the attack
> scenarios we should consider?
>
Not entirely sure, I suppose in the special case where one of the peers 
is the origin server you could do more?
>> -Automatic keep-alives (to detect force close at remote host and keep
>> NAT traversal active)
> I've left that up to the ICE layer.
>
>
>> -Reliable close handshake
> This can be done over the signaling layer independent of the UDP channel.
>
>
>> -Sockets open sequentially (like current DOS protection in WebSockets)
>> or perhaps have a limit of one socket per remote host.
>> -Cap on number of open sockets per host and global user-agent limit.
> UDP doesn't really have sockets, so I don't really know how to do this.
>
What about the ICE layer, is there anything that needs to be done to 
prevent flooding the server with requests there? I need to read up on 
ICE again.
>> Some additional points that were suggested on this list were: -Key
>> exchange and encryption If you do want to have key exchange and
>> encryption you really shouldn't reinvent the wheel but rather use a
>> secure WebSocket connection in addition to the UDP-WebSocket. Adding key
>> exchange and encryption to the UDP-WebSocket is discouraged.
> Not really sure what this means.
>
Just keep it simple and clean and people can implement what they want in 
JS atop that.
>
>> -Packet delivery notification to be a part of the API. Again this is
>> believed to be better left outside the UDP-WebSockets spec and
>> implemented in javascript if the application developer requires it.
>
> Agreed.
>
>
>
> On Fri, 11 Jun 2010, Erik M�ller wrote:
>>> I'd recommend doing some real-world testing for max packet size.
>>> Back when the original QuakeWorld came out it started by sending a
>>> large connect packet (could be ~8K) and a good number of routers would
>>> just drop those packets unconditionally.  The solution (iirc) was to
>>> keep all packet sends below the Ethernet max of 1500 bytes.  I haven't
>>> verified this lately to see if that's still the case, but it seems
>>> real-world functionality should be considered.
>> Absolutely, that's why the path-MTU attribute was suggested. The ~64k
>> limit is an absolute limit though at which sends can be rejected
>> immediately without even trying.
> Could you elaborate on this use case?
>
Like I said earlier, this might not be an issue today, but you used to 
be able to see packets over a certain size getting unconditionally 
dropped if some dodgy router happened to be on the path. If that's still 
an issue it would be useful to know what the real MTU for the path is. 
But again... perhaps it's better to keep the API simple and leave that 
up to users.
>>> If WebSocket supports an encrypted and unencrypted mode, why would the
>>> real-time version not support data security and integrity?
>> The reasoning was that if you do need data security and integrity the secure
>> websocket over TCP uses the same state-of-the-art implementation as the
>> browsers already have implemented. Secure connections over UDP would either
>> require a full TCP over UDP implementation (to use TLS) or a second
>> implementation that would need to be maintained. That implementation would be
>> either a very complex piece or software or clearly inferior to that users are
>> accustomed to.
>> So what's a good use-case where you want a secure connection over UDP and
>> cannot use a second TLS connection?
> Games, if you want to prevent some forms of cheating. I don't necessarily
> agree that we have to do anything as complex as TLS (or DTLS) though.
> Encrypting the data stream gets us a long way there; we can add some
> integrity protection and replay protection reasonably easily too. Since we
> have a (presumed secure) signaling channel, a lot of the complexity of
> (e.g.) DTLS is unnecessary.
>
Possibly yes, but my gut feeling is keep it simple. Any game being 
really serious about security is at least going to have you sign in to a 
secure login server before kicking any UDP connections off, so whatever 
encryption and shared secrets they need to make the UDP connection 
tamper proof can be negotiated over that connection.