[whatwg] Media queries, viewport dimensions, srcset and picture

[Florian Rivoal]

Sorry for not replying to the right message in the thread,
I was previously not subscribed to this list, so I can't.

As the editor of the CSS Media Queries spec, I've been asked
to share my opinion about this debate on responsive images, srcset,
media queries, etc.

Disclamer: I haven't followed the discussion too closely, and I haven't
really done my homework of reading everything that's been said so far,
so I'm sure I'll miss the point in a bunch of places, but here's my
brain dump, for what it's worth.



The responsive image problem is made significantly harder by the fact that
in most cases, the decision of whether you want to use a high res or low
res image depends both on the resolution of the device and on the network
bandwidth.

For a low res device, no matter what the bandwidth is, you're going to
want a low res image (at least as long as you don't take zooming into
account). For a high res device, you want a high res image, unless the
bandwidth would make it to slow to load, in which case you might prefer a
low res image. If you have a variable bandwidth, the last thing you want
to do is to change your mind about which resolution you wanted half way
through due to a change of bandwidth, and discard already downloaded data
because it's the wrong one after all. This situation gets worse as with
high latency.

Leaving syntax aside, I think media queries in relation to images can be
useful. Providing a different image based on aspect ratio, color depth,
viewport size, etc can certainly make sense.

But I am skeptical that media queries can solve the responsive image
problem well, because I don't see how one could make a bandwidth or
latency media query that doesn't cause already downloaded content to be
discarded when the conditions change, other than by making it not update
to reflect the current conditions, which would make it fairly useless.

Given a set of images of different qualities, browsers can have fairly
advanced heuristics to pick the right one. For example switching from low
res to high res halfway through the rendering of the page if the device is
high resolution and the bandwith just went from bad to good and the
latency is low enough. Most authors are not going to bother writing media
queries of that complexity, and as media queries are stateless, even if
they wanted they couldn't be as sophisticated as browsers. This gets even
more true if you consider zooming.

Having said all that, I think srcset="foo.jpg 1x, foo2.jpg 2x" is quite
good, because it does indeed provide the browser with a set of images with
different quality, leaving it free to pick the appropriate one.

On the other hand, I think that including 600w 400h in there is misguided.
It doesn't help for the problem of picking the right image for the right
resolution/bandwidth combination, but is too crippled to be useful as
media queries meant to serve different images to in different scenarios.
<picture> serves these use cases much better.

Here's what I think we should do:

1) simplyfy srcset to only accept the *x qualifier

2) add support for srcset as an attribute of the <source> sub-element of
the <picture> element (in addition to src, or instead of it? I am not
sure).

Then you could do stuff like this:
<picture>
<source media="(orientation:landscape)" srcset="long.jpg 1x, long2.jpg 2x">
<source media="(orientation:portrait)" srcset="tall.jpg 1x, tall2.jpg 2x">
<img src="fallback.jpg" />
</picture>

Note that it is different from:

<picture>
<source media="(orientation:landscape) and (resolution:1dppx)"
src="long.jpg">
<source media="(orientation:landscape) and (resolution:2dppx)"
src="long2.jpg">
<source media="(orientation:portrait) and (resolution:1dppx)"
src="tall.jpg">
<source media="(orientation:portrait) and (resolution:2dppx)"
src="tall2.jpg">
<img src="fallback.jpg" />
</picture>

because it allows the browser to be smart about loading the high or low
res image depending on the network conditions. The solution purely based
on media queries doesn't let you do that.

One final note: the "1x, 2x" ... solution still has one problem that I
think cannot be properly solved purely in html/css. Even though the
browser can be smart about which image to used based on network
conditions, it still cannot escape the fact that to change its mind half
way through, it will have wasted time downloading the wrong image. It
may be worth it in some cases, but it is still wasteful.

I believe the only way out is through an image format that:
- stores multiple resolutions in one file
- stores the higher resolution as incremental information on top of the
low res, so that downloading low res images isn't a waste of time even if
you want the high res.
- is designed so that the browser can stop downloading half way through
the file, if it determines it got sufficiently high resolution given the
environment.

This would allow browsers to switch from wanting a high res image to
wanting a low res image without having to restart the download from
scratch, which is really bad, as the main reason for switching from high
to low is a bad network. When the browser is aiming for the high res
image, it still gets some lower quality image to display temporarily while
the higher quality image is being downloaded.

I am not enough of an image guy to know if progressive jpeg or webp or
some other existing format has these characteristics.

The "1x, 2x..." syntax probably needs to be tweaked to accommodate such
images:
srcset="standard.jpg 1x, progressive.prog 0.1-4x"

Even if we don't have an existing format for that, the html syntax should
probably anticipate it, so that soon-to-be implemented UAs don't get
confused when they get served content from the longer term future that
uses this syntax.

As an aside, This syntax might work to mix raster images and vector
images: srcset="foo.svg 0.1-0.5x, foo.jpg 1x"