[whatwg] Features for responsive Web design

[Mathew Marquis]

On Oct 5, 2012, at 6:09 PM, Ian Hickson <ian at hixie.ch> wrote:

> 
> Some of the e-mails on this thread were cross-posted to multiple mailing 
> lists. Please remember not to cross-post when posting to this list.
> 
> On Wed, 5 Sep 2012, Fred Andrews wrote:
>>>> 
>>>> I have always been comfortable with the 'x' part of srcset, but the 
>>>> w and h part felt somewhat wrong to me. What you'd really want to 
>>>> consider when deciding which image to pick isn't the size of the 
>>>> viewport itself, but the size available for the image once the rest 
>>>> of the layout is taken into account.
>>> 
>>> Yeah. That's how I originally designed srcset="", actually, but it was 
>>> pointed out to me that that's impossible to implement because at the 
>>> time the browsers need to pick an image, they haven't yet gotten the 
>>> style sheet so they don't know what the layout will be.
>>> 
>>> (Note that the media-query-based solutions have the same problem.)
>> 
>> If people are really concerned about this latency then they can inline 
>> the style so that the image layout size is known before other resources 
>> are available
> 
> Browser vendors don't seem emenable to this design, unfortunately. 
> (Without good reason; it turns out that the people who care about latency 
> are more often the users than Web authors -- Web authors often do things 
> that are unnecessarily slow.)
> 
> 
>> - this may just be the image CSS pixel size and many of these proposals 
>> require this to be included anyway.  It will also help with backwards 
>> compatibility to have the style available.  For example:
>> 
>> <img style="width: 10em" src="image-320x200.jpg" set="image-320x200.jpg 
>> 320 200 10k, image-640x400.jpg 640 400 40k, image-1280x800.jpg 1280 800 
>> 150k">
>> 
>> The dimensions here are in image pixels, not CSS pixels.  The set would 
>> include the 'src' image to give the declared image pixel size. The byte 
>> size and perhaps height could be optional.
> 
> Inlining the style here doesn't help. You don't know how many pixels "em" 
> means at the time of parsing.
> 
> 
>> In other cases, browsers could either delay loading the image or lookup 
>> the 'src' image in the set to obtain the declared image pixel size and 
>> use this to speculatively load an image (once the image viewport size is 
>> finalized the browser could then decide if a higher resolution image is 
>> needed and load it then if necessary).  Browsers will need to be 
>> prepared to reload a higher resolution image anyway in case of zooming 
>> in.
> 
> Zooming in post-load is a rare case compared to the case of an image being 
> sized differently, though.
> 
> 
>>> On Wed, 22 Aug 2012, John Mellor wrote:
>> ...
>>>> So for example "1280.jpg 4x" means that this image is 4 times larger 
>>>> than the given intrinsic width of 320px. So sure, it would be 
>>>> suitable for display on a hypothetical 4x display at 320px width; 
>>>> but the browser also knows that it would be suitable for display on 
>>>> a 2x display at 640px width, a 1.5x display at 853px width, and a 1x 
>>>> display at 1280px width.
>>> 
>>> This isn't accurate. A trivial example of it not being accurate is a 
>>> 1000 device pixel image that consists of a horiontal double-headed 
>>> arrow labeled "Five Hundred CSS Pixels". That image is _only_ 
>>> applicable in a 500 CSS pixel double-density environment. If you sed 
>>> it in a 250 CSS pixel quad-density environment, it would be wrong.
>> 
>> This example may miss the point.  If an image is to be scaled to 500 CSS 
>> pixels then this can be specified independently of the image pixel 
>> dimensions.  The browser may even decide to download a much smaller 
>> image that is labeled "Five Hundred CSS Pixels" and scale it to the 
>> require CSS pixel size for the benefit of low bandwidth devices.
> 
> Having both features seems like a lot of complexity. It's not clear to me 
> that it is needed in practice.
> 
> 
>>>> The art direction use case can be entirely orthogonal. It should be 
>>>> handled with the w/h descriptors as currently specified. What I'm 
>>>> proposing would operate after any w/h descriptors have narrowed down 
>>>> the set of allowable images, and let the browser choose between the 
>>>> remaining images more intelligently in the case of flexible-size 
>>>> images, where currently the browser has no idea which to use.
>>> 
>>> Could you give a real example of this kind of thing? I'd love to study 
>>> what kinds of images we're talking about here. None of the examples 
>>> people posted when we were designing srcset="" were like this.
>> 
>> Here's an example of a webpage with images that scale and for which the 
>> aspect ratio of the image frame changes fluidly to fit the window width: 
>> http://london.msn.co.uk/
> 
> That's an interesting page design, thanks.
> 
> For this one you really want the image whose dot width is the _device_ CSS 
> pixel width multiplied by the device density, assuming you don't care 
> about zooming, but do care about resizing. If you care about zooming more 
> than latency, you really just want the highest-res image.
> 
> In practice you'd probably have a medium-size image for legacy cases, a 
> low-size image for phones, and a high-size image for high-density big 
> devices, and would invoke them as follows (ish):
> 
>   srcset="low.jpeg 400w 1x, medium.jpeg 800w 2x, medium.jpeg 1600w 1x, high.jpeg 2x"
> 
> ...or some such. I agree that it's suboptimal for this specific scenario, 
> I agree that it means there's an editing hazard, but it doesn't seem so 
> bad that we should redesign the feature or add more features to it at this 
> time. If it turns out to be so common that we should support it (easily 
> detected by seeing how many people are listing the same URL multiple times 
> in srcset=""), we can add support for this later.
> 
> 
>>>> Image cropping and centering is something best left to JS 
>>>> algorithms.  Fluid cropping and centering may need an algorithm that 
>>>> takes into account particular areas of interest and focus within an 
>>>> image, and user choices which would be better handled within the 
>>>> webpage and not at the browser level.  A web gallery may allow 
>>>> visitors to choose to scale versus crop images and to pan cropped 
>>>> image.
>>> 
>>> Actually panning and so on would presumably best be done using 
>>> <canvas>. I don't think we need to be providing a declarative way to 
>>> do this.
>> 
>> This does seem to conflict with the srcset proposals 'artistic' choice 
>> of image crops.
> 
> Those are trivial crops in comparison to visitors chosing to scale, etc.
> 
> 
>> Btw <canvas> is not necessary for panning and cropping, the image can be 
>> positioned through styling and this can be modified by JS based on 
>> buttons or mouse movement etc.
> 
> Yes, you can do that too.
> 
> 
> On Wed, 5 Sep 2012, Nils Dagsson Moskopp wrote:
>>> 
>>> I don't understand why it's more intuitive and easier. It seems way 
>>> more unwieldly.
>> 
>> Personally, I consider <picture> with <source> to be very similar to 
>> using ATOM <enclosure>s in podcasting. The relation – there are 
>> several sub-resources that represent (more or less) one logical resource 
>> – directly maps to a container element with other elements in it.
>> 
>> Having <source> elements would also allow to support future image 
>> formats while still having a fallback via content-type.
> 
> Introducing new image formats is so rare (once every 20 years or so, so 
> far) that I don't think we should optimise for it, certainly not at such 
> a high cost. There are existing solutions (e.g. <object>) for handling 
> that kind of thing.

Could you expand a bit more on the “cost” of an approach that might account for this?

> 
>>> Manipulating <picture> from script would be a huge pain -- you'd have 
>>> to be manipulating lots of elements and attributes.
>> 
>> Well, is manipulating <audio> or <video> from script a huge pain?
> 
> Yes.

Certainly not more so than manipulating strings, having done both frequently myself.

> 
>> I actually have one use case that would benefit from having separate 
>> elements instead of an attribute – replacing <source> elements with 
>> links to their content for accessability purposes. I did something like 
>> this when I hacked elinks to (badly) support HTML5 media elements 
>> <http://blog.dieweltistgarnichtso.net/html5-media-elements-in-elinks>.
>> 
>> Consider that any attribute microsynthax would introduce a burden on 
>> programmatic DOM manipulation, as the attribute would have to be parsed 
>> separately. „Do X for every <source> child element“ is cognitively 
>> cheap in comparison to maintaining a mental model of the attribute in 
>> question – different from other mental models used in HTML – in your 
>> working memory.
> 
> There are plenty of attributes with more complicated syntaxes, e.g. all 
> the event handler attributes (whose syntax is JavaScript), or style="" 
> (whose syntax is CSS). See also <meta content> for many of the pragmas, 
> the <area coords> attribute, <ins datetime>, media="" attributes, etc.
> 

Saying “there are worse things” doesn’t make much of a case for a worse thing. Better that we focus on finding the best possible approach here and avoid our previous mistakes.

>> Furthermore, introducing an API would not help the use case of just 
>> parsing HTML in, say, Python, to programatically download all images 
>> from a page (or something like that).
> 
> Sure, but for that case you just need to write a library once (just like 
> you only need to write an HTML parsing library once).
> 
> 
>> This reminds me that ATOM <enclosures> have a byte length. Surfing via 
>> mobile, I certainly know that I would like images to show if they can be 
>> downloaded in a reasonable time – but I want to skip 5MB photos.
> 
> Given that newer mobile networks are actually faster than the networking a 
> lot of people in the US have to their house, I don't know how much of a 
> lifetime such a feature would have.

This seems incredibly specific to privileged browsing contexts, and hardly a standpoint that accounts for the millions of users in developing countries accessing the Internet from mobile devices alone. Users with limited bandwidth and a per-kilobyte economic cost for access to a resource that—by very design—is meant to be open and accessible to users of any context. Tim Berners-Lee can speak to that far better than I ever could.

>>>> a) using the 'monocrhome' MQ to serve gray scale images to 
>>>> black-and-white printers or e-ink displays. Displaying a color image 
>>>> on a monochrome display does not always work well, as two different 
>>>> colors of similar luminosity would be impossible to distinguish in a 
>>>> monochrome environment. I expect this need to grow together with the 
>>>> increasing popularity of HTML based ebooks.
>>> 
>>> Is this a real use case or a theoretical one? Until we didn't support 
>>> it, nobody once mentioned that it was a use case they cared about -- 
>>> they only mentioned dimensions as being the issue.
>> 
>> There seem to be quite some web devices that use black-and-white epaper. 
>> In a world in which people optimize content for mobile, tablets and 
>> accessability, I would certainly consider this when making a site. 
>> <http://www.youtube.com/watch?v=zXZDn2Ia9js>
> 
> Are there any sites today that do this?
> 
> 
>>>> 3) you syntax is terser, which is in generally a good thing, but I 
>>>> think it crosses the limit, as a large number of people have 
>>>> expressed confusion as to w and h were min or max, for example. The 
>>>> extra verbosity of my syntax gets you an extra bit of clarity, 
>>>> admittedly at the cost of having multiple elements.
>>> 
>>> I agree that there's a small learning curve, but it seems pretty easy 
>>> to understand. Do we really want to trade the small learning curve for 
>>> a perpetuity of verbosity?
>> 
>> As a programmer using Python, I am would argue for the latter. If markup 
>> is easier to read and understand for humans, people make fewer errors. 
>> Certainly, in uncommon cases (I consider <p> a common case) verbosity is 
>> helpful for both learning and readability.
> 
> Well sure, the point is to strike the right balance.
> 
> 
>>> Fundamentally, a multiple-element solution here is simply a 
>>> non-starter, IMHO. The pros of the multielement solution with verbose 
>>> media queries are about the same in magnitude as the pros of the 
>>> one-attribute solution with terse syntax, but the cons of the terse 
>>> syntax are small whereas the cons of the multiple-element syntax are 
>>> immense. For the multi-element solution to be a net positive over the 
>>> one-attribute solution, the magnitude of its "pros" would have to be 
>>> enormous.
>> 
>> Does readability count?
> 
> It's not at all clear to me that the <picture> proposals are more 
> readable. It's certainly not an enormous enough difference to be relevant.

Perhaps, given the lack of clarity on this point, we might consult the opinion of authors.

> On Wed, 5 Sep 2012, Mathew Marquis wrote:
>>> 
>>> Whether it's easier for script is hard for me to say, because I don't 
>>> really understand what scripts are going to be doing here. Can you 
>>> elaborate? What will scripts need to do here?
>>> 
>>> If it is harder to script, we can always provide a dedicated API.
>>> 
>>> Manipulating <picture> from script would be a huge pain -- you'd have 
>>> to be manipulating lots of elements and attributes.
>> 
>> I can say for my own part: manipulating strings is far more difficult 
>> than manipulating the value of individual attributes. It’s hard to 
>> imagine a situation where I’d prefer to muck through a space/comma 
>> separated string rather than a set of independent elements and 
>> attributes. Unless the plan is to include an API similar to classList, 
>> though it would then be occupied by a set of strings describing 
>> disparate information.
> 
> If we provide an API, it would do whatever people need to do. First we 
> have to work out what people want to do. For example, if people need a way 
> to find the image that matches a particular viewport dimension and 
> density, then we'd provide a function like this:
> 
>   URL getMatchingImage(long width, long height, double density);
> 
> 
> On Wed, 5 Sep 2012, Miguel Garcia wrote:
>>>> 
>>>> Manipulating <picture> from script would be a huge pain -- you'd have 
>>>> to be manipulating lots of elements and attributes.
>> 
>> No more than the already accepted syntax and structure for video.
> 
> Exactly.
> 
> 
>> Not that one would really be manipulating tons of elements and 
>> attributes but swapping out sources for things like a photo gallery and 
>> the like are things that will happen. We should expect the general use 
>> cases for script manipulation that exist today for img will naturally 
>> migrate to picture if it indeed becomes the new standard.
> 
> I agree. Who manipulates <img>, though? Surely you just create the image 
> with the image you need, and use it. No manipulation involved. For 
> srcset="", it's at most a concatenation of a few strings. When would you 
> _parse_ it?

Should we site examples of sites that grab/manipulate the value of an img tag’s src? Speaking of bandwidth costs: that would make for an exceedingly long email.

> On Thu, 6 Sep 2012, Tab Atkins Jr. wrote:
>> 
>> The srcset attribute, as currently written, is not friendly to large 
>> screen-size differences that don't trigger different "art direction".
>> 
>> Consider the following example:
>> 
>> <img srcset="800.jpg 1x, 1600.jpg 2x" style="width: 100%; height: auto;">
>> 
>> For a screen that's somewhere near 800px wide, this works just fine. 
>> However, a 1x screen 1600px wide (not too uncommon - I think a 19" 
>> monitor is roughly that width) will get served the 800.jpg image, which 
>> then gets blown up to an unattractive level.  The 1600.jpg file should 
>> be identical to the 800.jpg file, just higher resolution, so delivering 
>> it instead would be ideal, but the current syntax doesn't allow that, 
>> nor does it allow any reasonably reliable way for a browser to detect 
>> that it would be okay to serve the 1600.jpg image either.
>> 
>> I'm not sure how best to solve this, but John Mellor suggested
>> allowing the specification of the image's native dimensions somehow.
>> That way, the browser could know that the 1600.jpg image is
>> appropriate to serve as an 800px wide high-dpi image, or a 1600px wide
>> low-dpi image.
>> 
>> It is possible to address this by repeating the same image at a larger
>> breakpoint, like:
>> 
>> <img srcset="800.jpg 1x 1599w, 1600.jpg 2x 1599w, 1600.jpg 1x">
>> 
>> However, this means you're duplicating data, and have a chance of
>> failing to update all of the urls when you update one.  It also
>> becomes more hostile as future screens arrive with higher resolutions.
>> For example, if 3x screens showed up, one would have to write the
>> following to serve things in the most ideal manner:
>> 
>> <img srcset="800.jpg 1x 1599w, 1600.jpg 2x 1599w, 2400.jpg 3x 1599w,
>> 1600.jpg 1x 2399w, 2400.jpg 1.5x 2399w, 2400.jpg 1x">
>> 
>> At this point it's just silly, and very error-prone.
> 
> I agree, when there's 3x displays, this could get to the point where we 
> need to solve it. :-)
> 
> With the current displays, it's just not that big a deal, IMHO.

Perhaps we should skate to where the puck will be, rather than where it is now.

> On Fri, 7 Sep 2012, Markus Ernst wrote:
>> 
>> I'd like to revive Florian Rivoal's (and my) suggestion of using 
>> <picture> for [...]
> 
> Multiple-element solutions really are non-viable, IMHO. Search for 
> "multple-element" in:
> 
>   http://lists.w3.org/Archives/Public/public-whatwg-archive/2012Aug/0070.html
> 
> -- 
> Ian Hickson               U+1047E                )\._.,--....,'``.    fL
> http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
> Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'