[whatwg] Proposal: downsample while decoding image blobs in createImageBitmap()

[Rik Cabanier]

Hi David,

is there a reason why you are completely decoding the image when you create
the imageBitmap? [1]
If you detect a situation where this operation causes excessive memory
consumption, you could hold on to the compressed data URL and defer
decoding until the point where it is actually needed.
Since exhausting VM will create "undue latency", this workaround follows
the spirit of the spec.

If you really want to have the downsampled bits in memory, you could create
a canvas and draw your image into it.

On Tue, Dec 17, 2013 at 4:31 PM, David Flanagan <dflanagan at mozilla.com>wrote:

> This is a proposal for changes to ImageBitmap and createImageBitmap()
> to enable the memory-efficient display of large images on small screens.
>
> Background:
>
> The camera resolution on mobile devices has grown (and is continuing
> to grow) much faster than the screen size and memory of those
> devices. In my work with FirefoxOS, I work with devices that have
> camera sensors that can capture 5 megapixels images but have 320x480
> pixel (0.15 megapixel) screens. This means that photos from the
> camera are 33 times as large as the screen.
>
> An RGBA image format requires 4 bytes per pixels for decoded image
> data, so if I want to decode one of these 5mp images for display on my
> 0.15mp screen, I have to allocate 20mb of memory. This particular
> low-end device I'm talking about has 256mb of RAM, and less than 200mb
> available for apps, so displaying a single photo requires more than
> 10% of available memory.
>
> To make this work in the initial releases of FirefoxOS, we've limited
> the camera resolution to 2 or 3mp on low-memory devices and have
> ensured that our Camera app includes screen-sized EXIF preview images
> in the photos it captures. We use JavaScript to extract the EXIF
> preview from the photo and display that, when we can, instead of the
> actual image. So initial display of a photo does not actually require
> us to decode the full-size photo. But as soon as the user zooms in,
> we have do have to decode it and take the memory hit. The result is
> that on low-end FirefoxOS phones background apps (including the
> homescreen) are commonly killed while using the Gallery app.
>
> The web platform has two mechanisms for decoding images: the <img>
> element and the new window.createImageBitmap() function. Native
> libraries exist that can downsample an image while decoding it, but
> the web platform does not expose this feature. This is a fundamental
> shortcoming: Web apps will not be able to achieve parity with native
> photo display and processing apps until they are able to decode and
> downsample a large image into a smaller bitmap.
>
> With that as background, I'd like to propose changes to the
> createImageBitmap() factory method and also related changes to
> ImageBitmap itself.
>
> Proposal:
>
> (Note createImageBitmap() can take any ImageBitmapSource object as its
> first argument. I am primarily concerned with case where the first
> argument is a Blob and image decoding is involved, though I think that
> the changes I am proposing will also work for the various other
> ImageBitmapSource types, but obviously no image decoding will be
> required for those other types.)
>
> 1) My main proposal is to add dw and dh (destination width,
> destination height) arguments to createImageBitmap() to specify the
> desired size of the resulting ImageBitmap. dw and dh could be
> restricted to be less than or equal to sw and sh, so that image
> enlargement is never required. Note that it is not necessary to
> specify that an implementation be memory efficient. Desktop
> implementations that are not memory constrained might decode in one
> step and then downsample in a separate step. It is enough here to have
> an API that allows memory-efficient implementations on devices that
> need it.
>
> 2) There are times when we want to downsample and decode the entire
> image rather than a region of it, and having a string of 6 optional
> arguments to createImageBitmap() is awkward, so I'd like to suggest
> that we convert the method so that it takes an ImageBitmapSource as
> the first argument and an options Dictionary as the second argument
> to hold the various optional (sx, sy, sw, sh, dw, dh, etc.) arguments.
>
> 3) Sometimes we want to decode and downsample a Blob but do not know
> the pixel size or aspect ratio of the original image, so we cannot
> specify exact dw, dh values.  My main use case here is to obtain a
> decoded image that is no bigger than necessary but maintains the
> aspect ratio of the original.  One way to get this would be to allow
> maxWidth and maxHeight properties in the options dictionary.  If those
> properties were defined, then createImageBitmap() would maintain the
> aspect ratio and create an ImageBitmap that is no wider than maxWidth
> and no higher than maxHeight.  Another, more flexible, solution would
> be to allow a size property in the dictionary. If size was omitted,
> then the dw and dh properties would be the actual size of the
> ImageBitmap, even if that resulted in distortion. If size was set to
> "contain", then the image would be downsampled to be as large as
> possible while still being contained within dw and dh and while
> preserving aspect ratio. (This is equivalent to the maxWidth and
> maxHeight properties).  And if size was "cover", then aspect ratio
> would be preserved, the resulting ImageBitmap would be exactly dw by
> dh pixels, but the image would be cropped along the top and bottom or
> the left and right edges to fit. Note that the names "cover" and
> "contain" come from the CSS background-size property.
>
> 4) Even when we downsample an image while decoding it, we may still
> need to know the full size of the original. In a photo gallery app,
> for example, we need to know how big the original is so that we know
> how far we can allow the user to zoom in to the image. It is possible
> to determine this by parsing the image file ourselves in JavaScript,
> but it would be much more convenient if the web platform provided a
> way to determine the full size of an image without having to decode
> the entire thing at the cost of 4 bytes per pixel.  Therefore I
> propose that the ImageBitmap include fullWidth and fullHeight
> properties that specify the full size of the ImageBitmapSource from
> which it was derived.  I suspect (but do not have an explicit use case
> for) that it might also be helpful to include the sx, sy,
> sw, and sh arguments that are passed to createImageBitmap on the
> returned ImageBitmap.
>
> 5) Once a large image is decoded and downsampled into a smaller
> ImageBitmap, the only thing that we can do with that ImageBitmap is to
> copy it into a Canvas, either for display to the end user (as an
> alternative to an <img>) or for re-encoding with Canvas.toBlob() (when
> creating thumbnails for large images). The motivation for this
> downsampling feature is memory use. But having to copy an ImageBitmap
> into a canvas in order to use it immediately doubles the amount of
> memory required. So for this reason, I also want to propose that
> ImageBitmap have a transferToCanvas() method akin to the
> transferToImageBitmap() and transferToImage() methods proposed at
> http://wiki.whatwg.org/wiki/WorkerCanvas.  transferToCanvas would
> transfer the image data into a new Canvas object and would neuter the
> ImageBitmap so that it could no longer be used.
>
> 6) Finally, because image data can take up so much memory, I would
> like to propose that ImageBitmap have a release() method to explicitly
> free the memory that holds the decoded image when that decoded image
> data is no longer needed. This gives web applications more precise
> control over memory allocation without having to wait for garbage
> collection.


There was an email thread on adding this to canvas [2], it seems reasonable
to add it to imageBitmap as well.


1:
http://www.whatwg.org/specs/web-apps/current-work/multipage/timers.html#dom-createimagebitmap
2: http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2013-July/040165.html