[whatwg] Video with MIME type application/octet-stream

[Ian Hickson]

Quick terminology point: in this e-mail, I use the term "sniff" to mean 
"examine the first few bytes of a resource, and determine its type 
heuristically" in contrast with assuming that the type of a file is that 
given by its MIME type (or, heaven forfend, the file extension).

On Thu, 20 May 2010, Robert O'Callahan wrote:
>
> I just became aware that application/octet-stream is excluded from being 
> a type "the user agent knows it cannot render". 
> http://www.whatwg.org/specs/web-apps/current-work/multipage/video.html#a-type-that-the-user-agent-knows-it-cannot-render 
> Apparently this was done in response to a bug report: 
> http://www.w3.org/Bugs/Public/show_bug.cgi?id=7977
> Neither the bug report nor the editor's response give any indication why 
> this change was made.

On Thu, 20 May 2010, Simon Pieters wrote:
> 
> Apparently this was done in response to:
> 
> http://lists.w3.org/Archives/Public/public-html/2009Jul/0888.html
> http://html5.org/tools/web-apps-tracker?from=3497&to=3498

On Thu, 20 May 2010, Robert O'Callahan wrote:
> 
> This change means files served with application/octet-stream will make 
> it all the way to the step "If the media data can be fetched but is 
> found by inspection to be in an unsupported format ...", so 
> implementations have to add support for binary sniffing for all the 
> types they support. We didn't need this before in Gecko. What was the 
> motivation for adding this implementation requirement?

On Thu, 20 May 2010, Robert O'Callahan wrote:
> 
> Hmm. I guess it doesn't add any implementation requirements beyond what 
> you need to handle the complete absence of a Content-Type (which we 
> currently don't handle, but I suppose we should). So this isn't really a 
> problem.

As Simon says, it was added based on a request from Mozilla engineers. :-)


On Thu, 20 May 2010, Philip Jägenstedt wrote:
> 
> For the record, Opera implements canPlayType("application/octet-stream") and
> canPlayType("application/octet-stream; codecs=foo") as per spec ("maybe" and
> "" respectively), but I don't have any strong opinions about it.

On Thu, 20 May 2010, Simon Pieters wrote:
>
> This bug report was about application/octet-stream *with parameters*, 
> e.g. application/octet-stream; codecs="theora, vorbis". The spec had the 
> requirement about application/octet-stream before that bug report.

On Thu, 20 May 2010, Simon Pieters wrote:
> 
> The spec requires binary sniffing for all the types you support even 
> without the application/octet-stream requirement, since a WebM file 
> labelled as video/ogg should play if both video/webm and video/ogg are 
> supported.

Currently that is indeed the case, but only Opera does it.

I spoke to various browser vendor engineers about this.

Microsoft's position is, as far as I can tell, that there's no point 
looking at the Content-Type header, so they always sniff the types of the 
video data once they've decided to try downloading it. I have no idea what 
algorithms they use, or how they would handle cases like Matroska (which 
doesn't have a guaranteed finite signature at the start of the file).

Safari does crazy things right now that we won't go into; for the 
purposes of this discussion we'll assume Safari can change.

Chrome right now sniffs like IE, modulo some bugs with text/plain that 
affect (only) the UI, but engineers tell me they're willing to change.

Opera does what the spec suggests. That is, if the type isn't supported 
then it is treated as a signal to give up without waiting for data to 
sniff; if the type _is_ supported then it isn't trusted, the data is 
obtained and examined to determine its real type.

Mozilla respects the Content-Type religiously, even if it gets data in a 
type it supports labelled with another type it supports, but engineers 
tell me they're willing to change.

As I see it we have two possible destinations: we can do what the spec 
says now, which is a somewhat reasonable (IMHO) compromise between 
religiously following the Content-Type, and being pragmatic about people 
getting the type wrong sometimes; or we can do what IE9 will do, and just 
always sniff once we've decided to try looking at the data, always 
ignoring the Content-Type headers. I think we all know where we're going 
to end up, but for now I've left the spec as is.


On Thu, 20 May 2010, David Singer wrote:
>
> Did anyone revise the registration of application/octet-stream to add 
> parameters?

On Thu, 20 May 2010, Simon Pieters wrote:
> 
> No. It's just error handling.

On Thu, 20 May 2010, David Singer wrote:
>
> It's an error to have a parameter that isn't valid for the mime type, so 
> are you suggesting (a) that you throw away the parameter as it's invalid 
> or (b) since it's an error to supply application/octet-stream as the 
> mime type in the first place, we may as well process its invalid 
> parameter in an attempt to recover?

On Thu, 20 May 2010, Simon Pieters wrote:
> 
> I'm just suggesting that it should be defined what to do when you get 
> application/octet-stream with parameters. I don't care which handling 
> that is, or whether it's valid or why the specific handling was chosen.

Agreed. Right now the spec says that it is considered not recognised (as 
if the type was application/utterly-bogus).


On Mon, 19 Jul 2010, Philip Jägenstedt wrote:
> 
> I've tested Firefox 3.6.4, Firefox 4.0b1 and Chrome 5.0.375.99 and none 
> return "maybe" for canPlayType("application/octet-stream"). I couldn't 
> get meaningful results from Safari on Windows (requires restart to 
> detect QuickTime, perhaps?).
> 
> It would appear that Opera is the only browser that supports 
> application/octet-stream. At the time I added this, it was simply 
> because it is true, maybe we can play it. However, I see no practical 
> benefit of this spec-wise or implementation-wise. Since no other 
> browsers have implemented it, I am going to remove it from Opera and 
> hope that the spec will be changed to match this.

Agreed. I've changed the spec to match.


On Tue, 20 Jul 2010, Robert O'Callahan wrote:
> 
> I actually think we should add support for application/octet-stream 
> media resources to Firefox, and if we're going to do that then I think 
> we should be consistent and return "maybe" for canPlayType too. So I 
> don't think you should remove this.

I think 'application/octet-stream' isn't a video type. The browser can't 
play it. It's just that it's ignored in certain cases.


On Wed, 21 Jul 2010, Philip Jägenstedt wrote:
> On Wed, 21 Jul 2010 15:55:24 +0200, Mike Shaver wrote:
> > On Wed, Jul 21, 2010 at 9:51 AM, Philip Jägenstedt wrote:
> > > On Wed, 21 Jul 2010 15:15:18 +0200, Mike Shaver wrote:
> > > > Could you be more specific about the incorrect information?  My 
> > > > understanding, from this thread and elsewhere, is that video 
> > > > formats are reliably sniffable, and furthermore that the 
> > > > appropriate MIME type for ogg-with-VP8 vs ogg-with-theora isn't 
> > > > clear (or possibly even extant).  It seems like reliance on MIME 
> > > > type will result in more of the guessing-and-stupid-switches than 
> > > > sniffing.
> > > 
> > > The MIME type for both of those would be video/ogg. It wouldn't be 
> > > hard or very error-prone to use only the MIME type, Firefox already 
> > > does that. It's also not very hard to rely on sniffing, which all 
> > > the other browsers do, although Opera still checks the MIME type 
> > > first.

It doesn't so much check it as do what the spec says, which is to just use 
it as another signal that the file isn't going to be understandable.


> > Indeed, so it seems that sniffing is always required, unless we expect 
> > reliable use of the "codecs" parameter to become widespread.  (I 
> > confess that I do not expect this, even if this group and the W3C 
> > exhort authors to do so.)
> 
> I can quite confidently say that the codecs parameter won't be used like 
> this. Even if it were always correct, setting up the media pipeline 
> isn't helped the slightest bit by it, as connecting the demuxer to the 
> decoder needs much more than just the types involved. At most one could 
> reject files that don't have the correct codecs parameter, but that 
> seems rather pointless.

Indeed. The spec doesn't say to use the codecs="" parameter to figure out 
what the file is -- just to use it to wean out the cases where the file is 
definitely not playable (because its type is something the player knows it 
doesn't know).


On Wed, 21 Jul 2010, Henri Sivonen wrote:
>
> There have been actual problems:
> 
> http://stackoverflow.com/questions/2643447/html5-video-mp4-and-ogv-problems-in-safari-and-firefox-but-chrome-is-all-good/2644341 

Would be solved by implementing the spec, since the spec hard-codes 
application/octet-stream as being a signal to sniff.


> http://stackoverflow.com/questions/2964193/html5-video-problem/2964458 
> http://krijnhoetmer.nl/irc-logs/whatwg/20100619#l-136 
> http://krijnhoetmer.nl/irc-logs/whatwg/20090807#l-795 
> http://krijnhoetmer.nl/irc-logs/whatwg/20100520#l-355 

I can't tell what types were being sent for these, so no idea if they 
would have been fixed by following the spec.


> http://twitter.com/hsivonen/status/14837369592

Would not have been solved. Could be solved by going the IE route or 
making text/plain magic here like application/octet-stream (we have 
precedent for that, sadly).


On Thu, 22 Jul 2010, Chris Double wrote:
> On Thu, Jul 22, 2010 at 6:10 AM, Boris Zbarsky <bzbarsky at mit.edu> wrote:
> > I don't like sniffing any more than the next guy, but the work needed 
> > to properly MIME label a modern media format (with the whole container 
> > and multiple streams thing) is ... excessive.  I doubt anyone's going 
> > to do it, so we're really talking about just labeling the container 
> > format, right?
> 
> Yes, in the majority of cases the MIME type will be for the container 
> format. We'd treat the MIME type much like canPlayType in that we'd try 
> to play any 'maybe' result from that MIME type I expect.

There are two ways to do that: the way Gecko does it today (try to play 
it by decoding it as the given type), and the way the spec and Opera do it 
(if it's 'maybe' or 'probably', then sniff).


> You might say "Hey, but aren't you content sniffing then to find the 
> codecs" and you'd be right. But in this case we're respecting the MIME 
> type sent by the server - it tells the browser to whatever level of 
> detail it wants (including codecs if needed) what type it is sending. If 
> the server sends 'text/plain' or 'video/x-matroska' I wouldn't expect a 
> browsers to sniff it for Ogg content.

The Microsoft guys responded to my suggestion that they might want to 
implement something like this with "what's the benefit of doing that?". 
It's a tough question, in this context, given that there's no possibilty 
of script execution or other privilege escalation with the types we're 
talking about (currently, anyway). "It's what the specs say" doesn't seem 
to be a particularly compelling argument for them.


> As I mentioned in a previous email, the sniffing could result in a 
> reasonable amount of data being consumed. I'm sure people who run sites 
> that share HTML 5 video would appreciate browsers not consuming data 
> bandwidth to sniff files that they've already specified as being 
> something the browser doesn't support.

Well, they can still use type="" on <source>, right? We're only talking 
about the case where the type is either missing on <source>, or there's 
only one src="", or the types on <source> is one that the user agent 
supports, in which case it seems reasonable to at least check that the 
server is not wrong after all. Also, sniffing the first few bytes is 
basically free, since they're highly likely to come in the same packet as 
the type anyway. 


On Thu, 22 Jul 2010, Philip Jägenstedt wrote:
> 
> Chrome and Safari ignore the MIME type altogether, in my opinion if we 
> align with that we should do it full out, not just by adding text/plain 
> to the whitelist, as that would either require (a) 
> canPlayType("text/plain") to return "maybe" or (b) different code paths 
> for checking the MIME type in Content-Type and for canPlayType.

On Thu, 22 Jul 2010, Maciej Stachowiak wrote:
> 
> I don't think canPlayType("text/plain") has to return "maybe". It's not 
> useful for a Web developer to test for the browser's ability to sniff to 
> overcome a bad MIME type. canPlayType should be thought of as testing 
> whether the browser could play a media resource that is "really" of a 
> given type, rather than labeled with that type over HTTP.

On Fri, 23 Jul 2010, Philip Jägenstedt wrote:
> 
> Right, it certainly isn't useful, I'm just pointing out that this is 
> what happens if one adds text/plain to the list of "maybe" codecs rather 
> than ignoring Content-Type altogether, which is the only thing you can 
> do within the bounds of the current spec to get text/plain to play. The 
> only 3 serious options I know are still the ones I outlined in my 
> earlier email.

canPlayType() is now hardcoded as not supporting application/octet-stream 
even though that type is otherwise not considered one that isn't supported 
(i.e. is a type that sniffs).


On Thu, 22 Jul 2010, johan wrote:
> 
> While the mime type discussion has been specific to video so far, as an 
> author I'd like to take a step back and look at it in a more general 
> way. First off, I'd expect handling of mime types to be consistent 
> across the board of all mime types, and more specifically, I'd expect 
> browsers to respect them. Wether it's text, image or video. For example, 
> if I serve a document, containing text that looks like an html document, 
> as text/plain, I'd expect the browser to serve it as text/plain.

For text/plain vs text/html it makes sense, and indeed in general for 
text/* types, I agree. However, for distinguishable binary types, what's 
the benefit?

It seems that sniffing is context-sensitive. When you're looking for an 
image, there are certain signatures to look for. When you're looking for 
an appcache manifest, there are other signatures to look for. It's only 
for some very specific types -- text/plain being the main one -- that you 
can't sniff the type without out-of-band metadata. For everything else, we 
could have included context-sensitive signatures.

Sadly, the boat has sailed for text/html and XML at this point, but for 
binary types, and for contexts where text/plain isn't a contender, why 
bother doing anything but sniff?


On Wed, 18 Aug 2010, Julian Reschke wrote:
> 
> From <http://greenbytes.de/tech/webdav/rfc2046.html#rfc.section.1>:
> 
> "Parameters are modifiers of the media subtype, and as such do not 
> fundamentally affect the nature of the content. The set of meaningful 
> parameters depends on the media type and subtype. Most parameters are 
> associated with a single specific subtype. However, a given top-level 
> media type may define parameters which are applicable to any subtype of 
> that type. Parameters may be required by their defining media type or 
> subtype or they may be optional. MIME implementations must also ignore 
> any parameters whose names they do not recognize."
> 
> So, as "codecs" is not defined on application/octet-stream, the 
> parameter simply should be ignored, thus the advice [...]:
> 
> "The MIME type "application/octet-stream" with no parameters is never a 
> type that the user agent knows it cannot render. User agents must treat 
> that type as equivalent to the lack of any explicit Content-Type 
> metadata when it is used to label a potential media resource.
> 
> Note: In the absence of a specification to the contrary, the MIME type 
> "application/octet-stream" when used with parameters, e.g. 
> "application/octet-stream;codecs=theora", is a type that the user agent 
> knows it cannot render."
> 
> is incorrect, because it requires handling "application/octet-stream" 
> and "application/octet-stream;codecs=theora" differently.

That's not incorrect. The type with no parameters is a special case that 
corresponds to a common configuration default. The case with parameters is 
not that case, and represents likely intentional configuration and thus 
clearly not a video format the UA supports.


> It's also not clear whether the note applies to all parameters or just 
> "codecs".

The normative text you quote doesn't mention any specific parameters.

Regarding codecs="" in particular, it's an implementation reality that 
user agents that support it are likely to support it regardless of the 
type, so there's really no point trying to maintain an artificial boundary 
of which types it has semantics for and which it doesn't.

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'