[whatwg] Referer header sent with <a ping>?

[Ian Hickson]

This e-mail is a response to all the recent ping="" feedback.

I carefully took into account all the feedback mentioned below, as well as 
past feedback and comments outside these mailing lists. In response to 
these comments, I've made Referer: be #PING for all pings, and added 
Ping-From and Ping-To to all pings that use the same origin or that aren't 
encrypted. I know this doesn't fulfill everyone's desires, but 
unfortunately the requests from people here were in some cases mutually 
exclusive and so it was not possible to please everyone. If I disagreed 
with something you said, either explicitly below or implicitly in the way 
the specification was changed, please do not take that as meaning your 
feedback was not welcome or not considered.


On Thu, 31 Jan 2008, Darin Fisher wrote:
> 
> Seems good to me.  It nicely addresses many of the concerns, and it also 
> makes <a ping> easier to use since you don't have to encode as much 
> information into the value of the ping attribute.
> 
> I suppose that X-Ping-From/To should be striped (like Referer) when one 
> of those values is HTTPS and the ping attribute is non-HTTPS?

If the From is SSL, then yeah, it'll only work if it's same origin. 
Doesn't much matter what we do with the To, though, if the source isn't 
encrypted, or if it's not the same origin as the from and ping, since the 
>From site can already do whatever it wants with that URI. Indeed, it came 
up with it.


On Fri, 1 Feb 2008, Anne van Kesteren wrote:
> 
> Given that HTML5 is now on standards track lets make it Ping-From and 
> Ping-To. Provisionally registering headers is pretty simple and when 
> HTML5 finally becomes a W3C Recommendation we can move them to the 
> permanent registry.

Done. Please do the honours of registering the headers. :-)


On Fri, 1 Feb 2008, Kornel Lesinski wrote:
> 
> Theoretically it does, but I haven't seen UA nor application that 
> supports it. Anyway, it could be made an URI with useless scheme, like 
> about:ping.

That could work, though really here we're trying to do something that's 
not a URI at all.


> Another advantage of headers is that Apache could log pings without help 
> of any scripts or non-standard modules - LogFormat directive allows 
> logging of arbitrary headers.

Indeed.


On Sat, 2 Feb 2008, Julian Reschke wrote:
> 
> How is that better compared not to send the Referer header at all?

As noted by others, no Referer header is treated as a local Referer 
header, which makes it susceptible to XSRF.


> > The point of it all is to make abuse of ping for CSRF harder, so 
> > standard body formats like www-form-urlencoded or XML are undesirable, 
> > but non-standard formats will require acceess to raw post data and 
> > custom parsers, which isn't as easy as reading headers.
> 
> So define a custom format.

As noted by Kornel, this is a significantly higher barrier to entry than 
just using headers.


> > Another advantage of headers is that Apache could log pings without 
> > help of any scripts or non-standard modules - LogFormat directive 
> > allows logging of arbitrary headers.
> 
> I'm not sure how this is relevant...

It seems extremely relevant, as it enables cheap server-side use instead 
of requiring heavy lifting for the author.


On Sat, 2 Feb 2008, Kornel Lesinski wrote:
> 
> Special Content-Type might work equally well -- it can be detected by 
> tools scanning headers only, and should prevent applications from 
> accepting unexpected POST.

Sadly I fear most sites don't check the Content-Type headers.


On Sat, 2 Feb 2008, Julian Reschke wrote:
> > 
> > Because not every client sends Referer, web applications have to 
> > accept requests without Referer at all. Bogus referer value avoids 
> > such whitelisting and can be easily blocked by anti-CSRF mechanisms.
> 
> So you want to abuse an HTTP/1.1 to implement blocking of ping requests. 
> That's really backwards. Instead, define the ping request in a way it 
> can be properly detected.

It's not a matter of abuse. We need to do a POST request, since that is 
the most appropriate method for this case (we don't want to add a PING 
method, just like we wouldn't want to add a LOGIN method, a SEARCH method, 
a POSTEMAIL method, a SAVEPREFERENCES method, etc). However, there is a 
minor risk with doing a POST request in a new way, which is that 
XSS blacklisting code wouldn't stop the new feature. This is a problem 
whether the Referer header is absent, or whether it contains a valid 
value. Hence, we have an invalid value. This seems emminently pragmatic, 
and not at all something I would classify as abuse.


On Sat, 2 Feb 2008, Julian Reschke wrote:
> > 
> > I see two ways forward here. One would be to redefine Referer to 
> > remove the relative URI thing, since, to my knowledge at least, nobody 
> > uses it.
> 
> That's IMHO not sufficient reason to remove it. It's not broken.

Lack of use is a reason to remove a feature according to IETF process, 
actually. It's even been used as a reason for HTTP -- Link: headers were 
removed from HTTP a while back for this very reason. (I've since worked 
hard to get browsers to implement it, and we can probably get it back in 
at this point. But that's another story.)

(RFC 2026, section 4.1.2.)


> > The other is that we can define the magic value to be "#PING" instead, 
> > since that's a non-conforming Referer value right now.
> 
> It's not conforming, so are you suggesting to use a non-conforming 
> value?

Right; by using a previously illegal value, we can ensure that nobody is 
relying on it already. (I presume your objection to using a relative value 
"ping" is that it could be misinterpreted by existing implementations due 
to its current definition.)


> Could you please state what problem you are trying to solve, and why it 
> needs to involve the Referer header?

I believe the problem has been stated a number of times in this thread 
already. I refer you to dolphinling's original e-mail.


> > The idea, as others have noted, is to keep the entity body empty so as 
> > to avoid any issues with servers that ignore the headers and process 
> > the body (which is relatively common).
> 
> Are you saying it wasn't a good idea to use POST after all because of 
> these risks?

Not at all; the risks are easily mitigated.


On Sat, 2 Feb 2008, Adam Barth wrote:
>
> Perhaps this has been suggested before, but another option is to use a 
> new verb, such as PING, instead of GET when making the request. Servers 
> unaware of the ping attribute will likely ignore this verb, mitigating 
> the request-forgery attack vector.

Sadly this is more than likely to cause problems with transparent proxies. 
It also seems silly to invent a new method, as noted earlier in this 
message. POST is the appropriate method here.


On Sat, 2 Feb 2008, dolphinling wrote:
> 
> If (X-)Ping-From/Ping-To are present, why is a referer needed at all? 
> I'd say just leave it out. If not, #PING works for me.

We can't remove it since then it would be treated as a local request.


> Cookies and authentication headers I'm ambivalent about; no one's made a 
> persuasive case either way for them, and I haven't looked myself.

I've required them, on the basis that removing them isn't solving any 
problems and including them makes it easier to accurately track users.


On Sun, 3 Feb 2008, Darin Fisher wrote:
> 
> It's true that the Ping-From/To headers carry the important information, 
> but I suspect that sending a Referer header is nice for servers that 
> happen to receive unsolicited pings.  Server logs will likely store the 
> value of the Referer header, and this way, site admins can see why they 
> are receiving these funny POST messages.

Good point.


> Julian, "#PING" seems like a reasonable choice since it will not collide 
> with any pre-existing Referer header values.  I think that's what Ian 
> had in mind by writing "non-conforming".

Indeed.


On Sun, 3 Feb 2008, Julian Reschke wrote:
> 
> I think it's a strange way to design a network protocol. When ping 
> requests in access logs are a problem, there are many ways not to have 
> it in the first place (not using HTTP, for instance) or by using a 
> different HTTP verb. Forcing something into a header it's not designed 
> for is even worse, in particular if the only reason given is to make it 
> easier to parse logs.

It's not clear exactly what your concern is here.


On Mon, 4 Feb 2008, Jon Barnett wrote:
> 
> The ability to log pings according to session data would be important to 
> me.  That can be done without cookies, but not very conveniently. I 
> guess the same applies to authentication headers, though I rarely use 
> them.  But yes, relating a ping back to a specific user is useful.

Ok.


> Since cookies and authentication headers are something I can do without 
> @ping, I think it would be a necessary feature to compel me to use 
> @ping.

Indeed.

Cheers,
-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'