[whatwg] Features for responsive Web design

[Ian Hickson]

Some of the e-mails on this thread were cross-posted to multiple mailing 
lists. Please remember not to cross-post when posting to this list.

On Wed, 5 Sep 2012, Fred Andrews wrote:
> > > 
> > > I have always been comfortable with the 'x' part of srcset, but the 
> > > w and h part felt somewhat wrong to me. What you'd really want to 
> > > consider when deciding which image to pick isn't the size of the 
> > > viewport itself, but the size available for the image once the rest 
> > > of the layout is taken into account.
> > 
> > Yeah. That's how I originally designed srcset="", actually, but it was 
> > pointed out to me that that's impossible to implement because at the 
> > time the browsers need to pick an image, they haven't yet gotten the 
> > style sheet so they don't know what the layout will be.
> > 
> > (Note that the media-query-based solutions have the same problem.)
> 
> If people are really concerned about this latency then they can inline 
> the style so that the image layout size is known before other resources 
> are available

Browser vendors don't seem emenable to this design, unfortunately. 
(Without good reason; it turns out that the people who care about latency 
are more often the users than Web authors -- Web authors often do things 
that are unnecessarily slow.)


> - this may just be the image CSS pixel size and many of these proposals 
> require this to be included anyway.  It will also help with backwards 
> compatibility to have the style available.  For example:
> 
> <img style="width: 10em" src="image-320x200.jpg" set="image-320x200.jpg 
> 320 200 10k, image-640x400.jpg 640 400 40k, image-1280x800.jpg 1280 800 
> 150k">
> 
> The dimensions here are in image pixels, not CSS pixels.  The set would 
> include the 'src' image to give the declared image pixel size. The byte 
> size and perhaps height could be optional.

Inlining the style here doesn't help. You don't know how many pixels "em" 
means at the time of parsing.


> In other cases, browsers could either delay loading the image or lookup 
> the 'src' image in the set to obtain the declared image pixel size and 
> use this to speculatively load an image (once the image viewport size is 
> finalized the browser could then decide if a higher resolution image is 
> needed and load it then if necessary).  Browsers will need to be 
> prepared to reload a higher resolution image anyway in case of zooming 
> in.

Zooming in post-load is a rare case compared to the case of an image being 
sized differently, though.


> > On Wed, 22 Aug 2012, John Mellor wrote:
> ...
> > > So for example "1280.jpg 4x" means that this image is 4 times larger 
> > > than the given intrinsic width of 320px. So sure, it would be 
> > > suitable for display on a hypothetical 4x display at 320px width; 
> > > but the browser also knows that it would be suitable for display on 
> > > a 2x display at 640px width, a 1.5x display at 853px width, and a 1x 
> > > display at 1280px width.
> > 
> > This isn't accurate. A trivial example of it not being accurate is a 
> > 1000 device pixel image that consists of a horiontal double-headed 
> > arrow labeled "Five Hundred CSS Pixels". That image is _only_ 
> > applicable in a 500 CSS pixel double-density environment. If you sed 
> > it in a 250 CSS pixel quad-density environment, it would be wrong.
> 
> This example may miss the point.  If an image is to be scaled to 500 CSS 
> pixels then this can be specified independently of the image pixel 
> dimensions.  The browser may even decide to download a much smaller 
> image that is labeled "Five Hundred CSS Pixels" and scale it to the 
> require CSS pixel size for the benefit of low bandwidth devices.

Having both features seems like a lot of complexity. It's not clear to me 
that it is needed in practice.


> > > The art direction use case can be entirely orthogonal. It should be 
> > > handled with the w/h descriptors as currently specified. What I'm 
> > > proposing would operate after any w/h descriptors have narrowed down 
> > > the set of allowable images, and let the browser choose between the 
> > > remaining images more intelligently in the case of flexible-size 
> > > images, where currently the browser has no idea which to use.
> > 
> > Could you give a real example of this kind of thing? I'd love to study 
> > what kinds of images we're talking about here. None of the examples 
> > people posted when we were designing srcset="" were like this.
> 
> Here's an example of a webpage with images that scale and for which the 
> aspect ratio of the image frame changes fluidly to fit the window width: 
> http://london.msn.co.uk/

That's an interesting page design, thanks.

For this one you really want the image whose dot width is the _device_ CSS 
pixel width multiplied by the device density, assuming you don't care 
about zooming, but do care about resizing. If you care about zooming more 
than latency, you really just want the highest-res image.

In practice you'd probably have a medium-size image for legacy cases, a 
low-size image for phones, and a high-size image for high-density big 
devices, and would invoke them as follows (ish):

   srcset="low.jpeg 400w 1x, medium.jpeg 800w 2x, medium.jpeg 1600w 1x, high.jpeg 2x"

...or some such. I agree that it's suboptimal for this specific scenario, 
I agree that it means there's an editing hazard, but it doesn't seem so 
bad that we should redesign the feature or add more features to it at this 
time. If it turns out to be so common that we should support it (easily 
detected by seeing how many people are listing the same URL multiple times 
in srcset=""), we can add support for this later.


> > > Image cropping and centering is something best left to JS 
> > > algorithms.  Fluid cropping and centering may need an algorithm that 
> > > takes into account particular areas of interest and focus within an 
> > > image, and user choices which would be better handled within the 
> > > webpage and not at the browser level.  A web gallery may allow 
> > > visitors to choose to scale versus crop images and to pan cropped 
> > > image.
> > 
> > Actually panning and so on would presumably best be done using 
> > <canvas>. I don't think we need to be providing a declarative way to 
> > do this.
> 
> This does seem to conflict with the srcset proposals 'artistic' choice 
> of image crops.

Those are trivial crops in comparison to visitors chosing to scale, etc.


> Btw <canvas> is not necessary for panning and cropping, the image can be 
> positioned through styling and this can be modified by JS based on 
> buttons or mouse movement etc.

Yes, you can do that too.


On Wed, 5 Sep 2012, Nils Dagsson Moskopp wrote:
> > 
> > I don't understand why it's more intuitive and easier. It seems way 
> > more unwieldly.
> 
> Personally, I consider <picture> with <source> to be very similar to 
> using ATOM <enclosure>s in podcasting. The relation – there are 
> several sub-resources that represent (more or less) one logical resource 
> – directly maps to a container element with other elements in it.
> 
> Having <source> elements would also allow to support future image 
> formats while still having a fallback via content-type.

Introducing new image formats is so rare (once every 20 years or so, so 
far) that I don't think we should optimise for it, certainly not at such 
a high cost. There are existing solutions (e.g. <object>) for handling 
that kind of thing.


> > Manipulating <picture> from script would be a huge pain -- you'd have 
> > to be manipulating lots of elements and attributes.
> 
> Well, is manipulating <audio> or <video> from script a huge pain?

Yes.


> I actually have one use case that would benefit from having separate 
> elements instead of an attribute – replacing <source> elements with 
> links to their content for accessability purposes. I did something like 
> this when I hacked elinks to (badly) support HTML5 media elements 
> <http://blog.dieweltistgarnichtso.net/html5-media-elements-in-elinks>.
> 
> Consider that any attribute microsynthax would introduce a burden on 
> programmatic DOM manipulation, as the attribute would have to be parsed 
> separately. „Do X for every <source> child element“ is cognitively 
> cheap in comparison to maintaining a mental model of the attribute in 
> question – different from other mental models used in HTML – in your 
> working memory.

There are plenty of attributes with more complicated syntaxes, e.g. all 
the event handler attributes (whose syntax is JavaScript), or style="" 
(whose syntax is CSS). See also <meta content> for many of the pragmas, 
the <area coords> attribute, <ins datetime>, media="" attributes, etc.


> Furthermore, introducing an API would not help the use case of just 
> parsing HTML in, say, Python, to programatically download all images 
> from a page (or something like that).

Sure, but for that case you just need to write a library once (just like 
you only need to write an HTML parsing library once).


> This reminds me that ATOM <enclosures> have a byte length. Surfing via 
> mobile, I certainly know that I would like images to show if they can be 
> downloaded in a reasonable time – but I want to skip 5MB photos.

Given that newer mobile networks are actually faster than the networking a 
lot of people in the US have to their house, I don't know how much of a 
lifetime such a feature would have.


> > > a) using the 'monocrhome' MQ to serve gray scale images to 
> > > black-and-white printers or e-ink displays. Displaying a color image 
> > > on a monochrome display does not always work well, as two different 
> > > colors of similar luminosity would be impossible to distinguish in a 
> > > monochrome environment. I expect this need to grow together with the 
> > > increasing popularity of HTML based ebooks.
> > 
> > Is this a real use case or a theoretical one? Until we didn't support 
> > it, nobody once mentioned that it was a use case they cared about -- 
> > they only mentioned dimensions as being the issue.
> 
> There seem to be quite some web devices that use black-and-white epaper. 
> In a world in which people optimize content for mobile, tablets and 
> accessability, I would certainly consider this when making a site. 
> <http://www.youtube.com/watch?v=zXZDn2Ia9js>

Are there any sites today that do this?


> > > 3) you syntax is terser, which is in generally a good thing, but I 
> > > think it crosses the limit, as a large number of people have 
> > > expressed confusion as to w and h were min or max, for example. The 
> > > extra verbosity of my syntax gets you an extra bit of clarity, 
> > > admittedly at the cost of having multiple elements.
> > 
> > I agree that there's a small learning curve, but it seems pretty easy 
> > to understand. Do we really want to trade the small learning curve for 
> > a perpetuity of verbosity?
> 
> As a programmer using Python, I am would argue for the latter. If markup 
> is easier to read and understand for humans, people make fewer errors. 
> Certainly, in uncommon cases (I consider <p> a common case) verbosity is 
> helpful for both learning and readability.

Well sure, the point is to strike the right balance.


> > Fundamentally, a multiple-element solution here is simply a 
> > non-starter, IMHO. The pros of the multielement solution with verbose 
> > media queries are about the same in magnitude as the pros of the 
> > one-attribute solution with terse syntax, but the cons of the terse 
> > syntax are small whereas the cons of the multiple-element syntax are 
> > immense. For the multi-element solution to be a net positive over the 
> > one-attribute solution, the magnitude of its "pros" would have to be 
> > enormous.
> 
> Does readability count?

It's not at all clear to me that the <picture> proposals are more 
readable. It's certainly not an enormous enough difference to be relevant.


On Wed, 5 Sep 2012, Mathew Marquis wrote:
> > 
> > Whether it's easier for script is hard for me to say, because I don't 
> > really understand what scripts are going to be doing here. Can you 
> > elaborate? What will scripts need to do here?
> > 
> > If it is harder to script, we can always provide a dedicated API.
> > 
> > Manipulating <picture> from script would be a huge pain -- you'd have 
> > to be manipulating lots of elements and attributes.
> 
> I can say for my own part: manipulating strings is far more difficult 
> than manipulating the value of individual attributes. It’s hard to 
> imagine a situation where I’d prefer to muck through a space/comma 
> separated string rather than a set of independent elements and 
> attributes. Unless the plan is to include an API similar to classList, 
> though it would then be occupied by a set of strings describing 
> disparate information.

If we provide an API, it would do whatever people need to do. First we 
have to work out what people want to do. For example, if people need a way 
to find the image that matches a particular viewport dimension and 
density, then we'd provide a function like this:

   URL getMatchingImage(long width, long height, double density);


On Wed, 5 Sep 2012, Miguel Garcia wrote:
> >> 
> >> Manipulating <picture> from script would be a huge pain -- you'd have 
> >> to be manipulating lots of elements and attributes.
> 
> No more than the already accepted syntax and structure for video.

Exactly.


> Not that one would really be manipulating tons of elements and 
> attributes but swapping out sources for things like a photo gallery and 
> the like are things that will happen. We should expect the general use 
> cases for script manipulation that exist today for img will naturally 
> migrate to picture if it indeed becomes the new standard.

I agree. Who manipulates <img>, though? Surely you just create the image 
with the image you need, and use it. No manipulation involved. For 
srcset="", it's at most a concatenation of a few strings. When would you 
_parse_ it?


On Thu, 6 Sep 2012, Tab Atkins Jr. wrote:
> 
> The srcset attribute, as currently written, is not friendly to large 
> screen-size differences that don't trigger different "art direction".
> 
> Consider the following example:
> 
> <img srcset="800.jpg 1x, 1600.jpg 2x" style="width: 100%; height: auto;">
> 
> For a screen that's somewhere near 800px wide, this works just fine. 
> However, a 1x screen 1600px wide (not too uncommon - I think a 19" 
> monitor is roughly that width) will get served the 800.jpg image, which 
> then gets blown up to an unattractive level.  The 1600.jpg file should 
> be identical to the 800.jpg file, just higher resolution, so delivering 
> it instead would be ideal, but the current syntax doesn't allow that, 
> nor does it allow any reasonably reliable way for a browser to detect 
> that it would be okay to serve the 1600.jpg image either.
> 
> I'm not sure how best to solve this, but John Mellor suggested
> allowing the specification of the image's native dimensions somehow.
> That way, the browser could know that the 1600.jpg image is
> appropriate to serve as an 800px wide high-dpi image, or a 1600px wide
> low-dpi image.
> 
> It is possible to address this by repeating the same image at a larger
> breakpoint, like:
> 
> <img srcset="800.jpg 1x 1599w, 1600.jpg 2x 1599w, 1600.jpg 1x">
> 
> However, this means you're duplicating data, and have a chance of
> failing to update all of the urls when you update one.  It also
> becomes more hostile as future screens arrive with higher resolutions.
> For example, if 3x screens showed up, one would have to write the
> following to serve things in the most ideal manner:
> 
> <img srcset="800.jpg 1x 1599w, 1600.jpg 2x 1599w, 2400.jpg 3x 1599w,
> 1600.jpg 1x 2399w, 2400.jpg 1.5x 2399w, 2400.jpg 1x">
> 
> At this point it's just silly, and very error-prone.

I agree, when there's 3x displays, this could get to the point where we 
need to solve it. :-)

With the current displays, it's just not that big a deal, IMHO.


On Fri, 7 Sep 2012, Markus Ernst wrote:
> 
> I'd like to revive Florian Rivoal's (and my) suggestion of using 
> <picture> for [...]

Multiple-element solutions really are non-viable, IMHO. Search for 
"multple-element" in:

   http://lists.w3.org/Archives/Public/public-whatwg-archive/2012Aug/0070.html

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'