[whatwg] Proposal: downsample while decoding image blobs in createImageBitmap()

[Rik Cabanier]

On Tue, Dec 17, 2013 at 11:59 PM, David Flanagan <dflanagan at mozilla.com>wrote:

>  On 12/17/13 10:55 PM, Rik Cabanier wrote:
>
>
>
>
> On Tue, Dec 17, 2013 at 9:36 PM, David Flanagan <dflanagan at mozilla.com>wrote:
>
>>  On 12/17/13 8:36 PM, Rik Cabanier wrote:
>>
>> Hi David,
>>
>>  is there a reason why you are completely decoding the image when you
>> create the imageBitmap? [1]
>>
>>  I assume that that is the intent of calling createImageBitmap() on a
>> blob. Since JPEG decoding probably takes significantly longer than blocking
>> on memory access, I assume that lazy decoding is not really allowed.
>>
>
>  No, nothing in the spec says that you *must* decode the bits:
>
>  The exact judgement of what is undue latency of this is left up to the
> implementer, but in general if making use of the bitmap requires network
> I/O, or even local disk I/O, then the latency is probably undue; whereas if
> it only requires a blocking read from a GPU or system RAM, the latency is
> probably acceptable.
>
>
>  In your case, things are reversed. Allocating system ram will kill
> performance and cause undue latency. Reading the JPEG image on the fly from
> a Flash disk will be less disruptive and faster.
>
>
>>  But that misses my point. On the devices I'm concerned with I can never
>> completely decode the image whether it is deferred or not.  If I decode at
>> full size, apps running in the background are likely to be killed because
>> of low memory. I need the ability to do the downsampling during the
>> decoding process, so that there is never the memory impact of holding the
>> entire full-size image in memory.
>>
>>
>>   If you detect a situation where this operation causes excessive memory
>> consumption, you could hold on to the compressed data URL and defer
>> decoding until the point where it is actually needed.
>> Since exhausting VM will create "undue latency", this workaround follows
>> the spirit of the spec.
>>
>>  If you really want to have the downsampled bits in memory, you could
>> create a canvas and draw your image into it.
>>
>>  I can't do that because I don't have (and cannot have) a full-size
>> decoded image.  I've got a blob that is a JPEG encoded 5 megapixel image.
>> And I want to end up with a decoded 320x480 image.  And I want to get from
>> A to B without ever allocating 20mb and decoding the image at full size
>>
>
>  The downsampling happens *during* the drawimage of the imageBitmap into
> the canvas. At no point do you have to allocate 20mb.
>
>
>   Ah. I see what you're saying now.  My first reaction was "that's
> brilliant!".  Unfortunately, my second reaction was that drawImage() would
> then block on the image decoding, and unless this was being done in a
> Worker, I'm almost certain that would be an unacceptable performance hit.
> (One of my use cases is scanning an SD card for hundreds of images and
> generating thumbnails for each of them.  If doing this used drawImage()
> calls that blocked the main thread, my UI would be completely
> non-responsive.)
>
> I also suspect that adding an async version of drawImage() to the canvas
> API is a non-starter because that API is pretty fundamentally synchronous.
>

It doesn't have to be synchronous. (ie Chrome draws everything asynchronous
) However, presenting the data will still block until all the draw calls
are completed so it wouldn't help in this situation.
Is createImageBitmap going to do the decoding in another thread and fulfill
the promise on the main thread?

Your workflow sounds like it should run in a worker to avoid blocking
behavior.
Is it too much work to add my proposal and the needed JavaScript objects to
a worker? (The browser could be smart about when to decode or hold on to
the blob by looking at available system memory)
If so, your proposal sounds reasonable. Could you also add support so the
type of downsampling can be specified? [1]

So for this reason, I also want to propose that
> ImageBitmap have a transferToCanvas() method akin to the
> transferToImageBitmap() and transferToImage() methods proposed at
> http://wiki.whatwg.org/wiki/WorkerCanvas.  transferToCanvas would
> transfer the image data into a new Canvas object and would neuter the
> ImageBitmap so that it could no longer be used.


We will have to define the CORS issues here.
I assume the canvas gets the same origin as the imageBitmap? The current
spec for imageBitmap only seems to allow same-origin which seems
limiting... [2]

1:
http://www.whatwg.org/specs/web-apps/current-work/multipage/the-canvas-element.html#dom-context-2d-imagesmoothingenabled
2:
http://www.whatwg.org/specs/web-apps/current-work/multipage/timers.html#images