[whatwg] Rich Paste & DataTransfer / DataTransferItems API

[Daniel Cheng]

I'm currently 60-75% complete landing the patches for image paste support in
Chrome. I've chosen to expose image/png instead of a raw bitmap through
event.clipboardData.items in Chrome as a Blob. DataTransferItem::getAsFile()
is currently specced to return a File; in my local changes, it secretly
returns a Blob instead. I think it may make sense to rename it to
getAsBlob() and change the corresponding add() method in DataTransferItems
to take a Blob. I think these basic changes should support the use case you
want if it gets more widely implemented.

Daniel

On Mon, Mar 14, 2011 at 15:39, Jeb Boniakowski <jebjeb at gmail.com> wrote:

> I would like to be able to paste images from the system clipboard into
> web apps.  E.g. an annotated screengrab right into the bug tracker, a
> pic into emails, an image from twitpic into a blog CMS's post screen.
>
> (Apologies if this has been discussed recently and conclusions made.
> I searched the archives and couldn't find anything.  Also, this email
> recapitulates a lot of stuff that will be familiar to participants in
> this list, but I'm hoping that by posting a summary of the state of
> affairs and then hopefully having errors/misconceptions pointed out,
> it will serve as good archive fodder for future people like me.)
>
> This is of course a special case of richer interaction with the system
> clipboard, but I think its a very special case, for two reasons:
>  - The general case of richer paste is fraught with peril: what the is
> your webapp supposed to do with pointers to various system data
> structures?
>  - The second-most common type of data, after text, manipulated on the
> web is images.  They are a huge part of people's interaction with the
> browser.  People move tons of images all over the web all the time,
> and having to bounce off the local filesystem for everything is a
> pain.
>
> Right now, there are effectively two intertwined APIs in the spec that
> relate to being able to paste images:
>  - The Old API, Microsoft style, which includes
> DataTransfer.getData(format)
>  - The New API: DataTransferItems, etc.
>
> -=The Old API=-
>
> With the Old API, the spec reads as if it could be possible to get
> this data, should the browser feel like giving it to you.  There's
> nothing that specifically limits what data the UA can give you, and it
> sounds like you should be able to ask for whatever the UA tells you it
> has: http://dev.w3.org/html5/spec/Overview.html#dom-datatransfer-getdata
> (note: there seems to a small typo in the current Editor's Draft: the
> getData() method only takes 'format', not a second parameter called
> 'data', right?  That's what the summary tables above say.  In either
> case, these should probably agree).
>
> There's a range in what the browsers return for types.  In Safari
> (r80833), having a png on the clipboard shows you:
>  - com.apple.pasteboard.promised-file-url
>  - public.tiff
>  - NSPromiseContentsPboardType
>  - com.apple.webarchive
>  - public.utf8-plain-text
>  - dyn.ah62d4rv4gu8yc6durvwwaznwmuuha2pxsvw0e55bsmwca7d3sbwu
>  - text/uri-list
>  - Apple files promise pasteboard type
>  - application/x-webarchive
>  - dyn.ah62d4rv4gu8y6y4usm1044pxqzb085xyqz1hk64uqm10c6xenv61a3k
>  - dyn.ah62d4rv4gu8zs3pcnzme2641rf4guzdmsv0gn64uqm10c6xenv61a3k
>  - WebURLsWithTitlesPboardType
>  -
> dyn.ah62d4rv4gu8yc6durvwwa3xmrvw1gkdusm1044pxqyuha2pxsvw0e55bsmwca7d3sbwu
>  - CorePasteboardFlavorType 0x75726C6E
>  - CorePasteboardFlavorType 0x75726C20
>  - text/plain
>  - public.url-name
>  - NeXT RTFD pasteboard type
>  - public.url
>  - com.apple.flat-rtfd
>  - com.apple.pasteboard.promised-file-content-type
>  - image/tiff
>
> Whereas the Chrome I have (10.0.648.133) shows me:
>  - text/html
>  - text/uri-list
>  - url
>
> Firefox (3.6.15):
>  - text/x-moz-url
>  - text/x-moz-url-data
>  - text/x-moz-url-desc
>  - text/uri-list
>  - text/_moz_htmlcontext
>  - text/_moz_htmlinfo
>  - text/html
>  - text/plain
>
> I don't think this contravenes the spec, except that for all of those
> rich types in Safari, you don't actually get anything when you ask for
> it.  The spec doesn't really mandate anything that you have to do with
> any specific types of data on the clipboard, which especially makes
> sense when considering data that originates outside the browser.
>
> This has one interesting implication though for data that originates
> inside the browser: drag/drop or paste with images loaded from data
> URIs.  Currently, the clipboard behavior of these is interesting but
> varies:
>  - Chrome will return an img tag with the data uri intact, with the
> Base64-encoded data for type "text/html" (preceded by a meta tag
> telling you you're getting html)
>  - Firefox does pretty much the same thing, minus the meta tag
>  - Safari doesn't present a "text/html" type in this case.  However,
> it does provide a "text/plain", which contains *just* the data uri
> itself.
>
> Now, this situation isn't preventing anyone from doing anything, as
> far as I can tell, but would it be crazy to try to normalize this in
> the HTML5 spec? Decide whether a dragged/pasted data uri-backed <img>
> counts as HTML or or text or neither, and when it does, what the HTML
> representation is appropriate to hand back?  For what its worth, FF
> and Chrome both return the plain data uri Safari-style if you ask for
> text/uri-list.
>
> As far as I can tell, there is little interest in expanding the Old
> API, since its an acknowledged hack-pile, but just to enumerate
> options, I think it would be cool if there was a way to get image data
> from here.  The obvious ways to do this would be:
>  - actually expose mime types like image/tiff and return a String of 1s and
> 0s.
>  - actually expose mime types, but when you ask for a binary one, you
> get a Blob.
>  - expose base64-encoded mime types.  I realize it seems a bit weird
> to push base64 encoding into the clipboard logic, but there are
> already several places in the current spec and implementation where
> base64-encoded binary data is given special treatment which provides
> both (a) precedent and (b) use cases.  For example, many of the
> browsers can load resources from base64-encoded data uri's.  The
> Canvas object exposes a getDataUrl() method that returns a
> base64-encoded png of the Canvas.  Having base64 data quick 'n' easy
> to get here would make it easy to push this data around in useful ways
> within apps.
>
> My guess is that no one wants to either add this to the current
> implementations, even to accept patches that implement this behavior,
> and that no one will want to add anything new to this part of the
> spec, given the focus on the newer area.  Also, requiring support for
> handling of specific binary formats seems to go against HTML's
> heritage.  So while spec-wise, there's nothing to keep implementors
> from adding application base64 versions of various image types to this
> API, and returning same from getData(), they probably won't, and it
> seems...unlikely to me that the spec will all of the sudden require
> say four image types here.
>
> So, moving on...
>
> -=The New API=-
> By the New API, I'm referring to the items member of DataTransfer,
> DataTransferItems, and DataTransferItem.
>
> Briefly, it looks to me like the use cases were more thought out here
> for interacting with system drag-drop than with copy-paste.  Excuse me
> if that isn't the case, and the idea of richer copy-paste was just
> considered to be out of scope.
>
> These features aren't supported by any of the shipping browsers I
> looked at, but it looks like someone has started implementing them in
> Chromium, there's talk on Firefox's list, and I'm not sure about
> Safari/don't know how/if Chromium patches work their way up and down
> into WebKit.  I haven't looked at the other browsers yet.  The point
> is, I haven't actually been able to play with this, so I'm just going
> by the spec here.
>
> According to the spec, with this API implemented, there will be no way
> to copy an image from a tab, and paste it into another tab (assuming
> the browser itself doesn't do something crazy like...bounce the system
> clipboard to a temporary file on paste and give you the filename
> somewhere).  The DataTransferItem can be of two kinds: a string, or a
> string referring to a file, so this lets you drag String-y data off
> your system and into a tab, and it also lets you presumably use
> Drag/Drop instead of the system's file browser to attach files.
>
> Is there any chance of adding something like a getBlob() method here?
>
> My initial thought about the trade-offs here vs. something like adding
> Base64'd binary types to the old DataTranfer.getData() is that its
> more in keeping with the overall direction of HTML5, but also more
> wrapped up in additional new APIs and less easy to immediately use.
>
> However, given the way system clipboards generally work, its pretty
> easy to imagine how you could:
>  - Map from certain system types to MIME types (or in some cases you
> already have MIME types handy, and they don't necessarily have to be
> in the spec)
>  - Add more types to DataTransferItems to show these (again, don't
> HAVE to be in spec)
>  - have a DataTransferItems.getBlob(format) that returns a Blob.
>
> For small objects you could handle this immediately or fail with a
> "too big" error, and with Web Workers you could handle larger pastes
> without blocking the UI thread the entire time.
>
> Again, sorry if this was discussed in the current cycle and punted on
> already, maybe I'm using the wrong search terms. Any thoughts about
> changing the spec so that it becomes possible to paste images?
>
> Thanks,
> jeb.
>