[whatwg] New URL Standard

[Boris Zbarsky]

On 9/24/12 4:58 AM, Anne van Kesteren wrote:
> Say you have <a href="data:test"/>; the concern is what e.g.
> a.protocol and a.pathname would return here. For invalid URLs they
> would return ":" and "" respectively. If we treat this as a valid URL
> you would get "data:" and "test". In Gecko I get "http:" and "". If I
> make that <a href="data:text/html,test"/> Gecko will give meaningful
> answers (well pathname is still "", maybe that is okay and pathname
> should only work for hierarchical URLs).

Ah, I see.

So what happens here is that Gecko treats this as an invalid URL (more 
precisely, it cannot create an internal "URI" object from this string). 
  I guess that's what you were getting at: that data: URLs actually have 
a concept of "invalid" in Gecko.  This is actually true for all schemes 
Gecko supports, in general.  For example, "http://something or other" 
(with the spaces) will do the same thing.

For an invalid URI, .protocol currently returns "http:" in Gecko.  I 
have no idea why, offhand.  It could just as easily return ":".

As far as .pathname, what Gecko does is exactly what you say: .pathname 
only works on hierarchical schemes.

> More general, what I want is that for *any* given input in <a
> href="..."/>, xhr.open("GET", ...), new URL(...), etc. I want to be
> able to tell what the various URL components are going to be. The kind
> of predictability we have for the HTML parser, I want to have for the
> URL parser as well.

Yes, absolutely agreed.

> (If that means handling data URLs at the layer of the URL parser
> rather than a separate parser that goes over the path, as Gecko
> appears to be doing, so be it.)

We could change Gecko's handling here, for what it's worth.  One reason 
for the current handling is that right now we don't even make <a> into a 
link unless its href is a valid URI as far as Gecko is concerned.  But 
I'm considering changing that anyway, since no one else bothers with 
such niceties and they complicate implementation a bit...

>> If you want constructive advice, it would be interesting to get a full list
>> of all the weird stuff that UAs do here so we can evaluate which parts of it
>> are needed and why.  I can try to produce such a list for Gecko, if there
>> seems to be motion on the general idea.
>
> I think that would be a great start. I'm happy to start out with
> Gecko's behavior and iterate over time as feedback comes in from other
> browsers.

Hmm.  So here goes at least a partial list:

1)  On Windows and OS/2, Gecko replaces '\\' with '/' in file:// URI 
strings before doing anything else with the string when parsing a new 
URL.  That includes relative URI strings being resolved against a 
file:// base.

2)  file:// URIs are parsed as a "no authority" URL in Gecko.  Quoting 
the IDL comment:

35     /**
36      * blah:foo/bar    => blah:///foo/bar
37      * blah:/foo/bar   => blah:///foo/bar
38      * blah://foo/bar  => blah://foo/bar
39      * blah:///foo/bar => blah:///foo/bar
40      */

where the thing on the left is the input string and the thing on the 
right is the normalized form that the parser produces from it.  Note 
that this is different from how HTTP URIs are parsed, for all except the 
item on line number 38 there.

3)  Gecko does not allow setting a username, password, hostname, port on 
an existing "no authority" URL object, including file://.  Attempts to 
do that throw internally; I believe for web stuff it just becomes a no-op.

4)  For "no authority" URLs, including file://, on Windows and OS/2 
only, if what looks like authority section looks like a drive letter, 
it's treated as part of the path.  For example, "file://c:/" is treated 
as the filename "c:\".  "Looks like a drive letter" is defined as "ASCII 
letter (any case), followed by a ':' or '|' and then followed by end of 
string or '/' or '\\'".  I'm not sure why this is checking for '\\' 
again, honestly.  ;)

5)  When parsing a "no authority" URL (including file://), and when item 
4 above does not apply, it looks like Gecko skips everything after 
"file://" up until the next '/', '?', or '#' char before parsing path stuff.

6)  On Windows and OS/2, when dynamically parsing a path for a "no 
authority" URL (not sure whether this is actually web-exposed, fwiw...) 
Gecko will do something involving looking for a path that's only an 
ASCII letter followed by ':' or '|' followed by end of string.  I'm not 
quite sure what that part is about...  It might have to do with the fact 
that URI objects in Gecko can have concepts of "directory", "filename", 
"extension" or something like that.

7)  When doing URI equality comparisons, if two file:// URIs only differ 
in their directory/filename/extension (so the actual file path), then an 
equality comparison is done on the underlying file path objects.  What 
this means depends on the OS.  On "Unix" this is just a straight-up byte 
by byte compare of file paths.  I think OS X now follows the "Unix" code 
path as do most other supported platforms.  But note that "file path" in 
this case is normalized in various ways.  Specifically: trailing '/' are 
stripped and some sort of normalization of HFS paths (possibly with a 
volume name) to POSIX paths is done on OSX.  One result of the latter is 
that file:///Users%2fbzbarsky ends up seeing my home directory, which is 
... slightly surprising.  On "Unix", the path bytes are treated as UTF-8 
if they're valid UTF-8, else treated as whatever the current locale 
charset is, I think.  Oh, and there is some sort of escaping going on 
for directory names, filenames, extensions.  Not sure what that's about, 
if anything.  The URI-escaping code is black magic, but I'm happy to run 
some black-box tests on it if someone wants to provide test strings.

The things that don't go through the "Unix" code for this stuff are 
Windows and OS/2.  I'm not going to dig through the OS/2 stuff, but on 
Windows if the filename contains a nonempty directory name and the 
second char is '|' that's converted to a ':'.  Again, escaping for 
directory names and file names and extensions.  Again, things that look 
like UTF-8 are treated thus and other stuff uses the current codepage. 
After all that, the actual equality comparison is done via _wcsicmp on 
the return value of GetShortPathNameW.  So whatever things that 
combination considers equal are equal.

8)  When actually resolving a file:// URL, the underlying file path 
object as described above is used to get the data.  Plus there's a bit 
of weirdness about symlinks, I think...  Mostly affects what's shown in 
the url bar when pointing the browser to a symlink.

That's what I can spot offhand.  I won't guarantee there is nothing 
else.  :(

-Boris